VR’s potential as a tool for immersive journalism is clearly immense. But there are also risks, as New York Times public editor Margaret Sullivan pointed out in a recent column. In particular, we need to make sure to keep the “reality” part of the term “virtual reality,” and to think carefully about what that means from a journalistic perspective.

Judging by some of the comments the Times has received on its Displaced project so far, a number of people were touched by the feature in a way they likely wouldn’t have been by a traditional photo series in the printed newspaper. And that’s probably not surprising, given the nature of the technology.

Instead of simply looking at photos or reading text, VR allows a viewer to experience a piece of journalism from the inside. This is done through the use of 360-degree video and/or virtual re-enactments of a scene broadcast onto a headset like the Oculus Rift (which is owned by Facebook (FB)), or even using low-tech means like the Google (GOOG) Cardboard viewer, which essentially just holds a smartphone in front of the watcher’s eyes.

VR can significantly change the way a person looks at a story, since they are more likely to feel as though they are part of it, rather than just a passive observer. Obviously, this kind of empathy can be a tremendously powerful tool. But it can also pose a risk to the traditional journalistic perspective of objectivity.

Sullivan discussed this point in her column, by responding to concerns raised by a number of readers. One such alarm was raised by Robert Kaiser, the former managing editor of The Washington Post, who said he was worried about the risk that stories told through VR “will often be based on tricks and deceptions by photographers/cameramen.”

In other words, because VR lets reporters, photographers, and videographers create a simulated version of a specific event, it could easy let them distort what happened and give a false impression of that event. That impression, in turn, would become more powerful due of the nature of the immersive technology.

Think about a VR version of the night that Michael Brown was shot by a police officer in Ferguson, Mo. The shooting quickly became a highly-charged political event, and one in which there were multiple versions of what happened—many of them completely contradictory. A virtual or immersive-reality feature on this story could easily distort the way that people feel about what happened.

Nonny de la Peña, an L.A.-based journalist and pioneer in VR film-making, created a 3D virtual representation of the night that Trayvon Martin was killed, and has also made an immersive film focused on the refugee crisis in Syria. She has talked about the risks of manipulating viewers through VR, and how the immersive nature of the experience can heighten this risk.

Some kinds of VR can lessen this problem, by providing multiple viewpoints from which a user can choose to view an event — something that Fusion’s version of the story tried to do. But other kinds of VR can reinforce the problem by giving a viewer only a specific version of the events (Tom Kent of the Associated Press has written about some of these ethical challenges).

Jake Silverstein, who was in charge of the New York Times project, told Sullivan he and his team were aware that there are a “whole host of ethical considerations and standards issues that have to be grappled with.” The paper’s standards editor, Philip Corbett, went through the piece frame by frame to make sure it fairly represented reality, and even recommended that a specific scene be removed because the photographer “was too active in re-arranging things,” he told Sullivan.

As Sullivan’s piece also pointed out, traditional video journalism has had to struggle with similar kinds of issues over the years, such as the use of “B-roll,” footage that is often shot after a report has already been researched, B-roll will typically show the scene, a subject walking towards the camera, or a reaction shot of the journalist. To some extent, these are “faked” shots, but they are widely used on television.

Ultimately, journalists and audiences must come to some kind of agreement on what is acceptable in the new world of immersive and VR journalism, and determine what kinds of film-making behavior are too manipulative. And the more we talk about that debate as it occurs, the better off we will be.

You can follow Mathew Ingram on Twitter at @mathewi, and read all of his posts here or via his RSS feed. And please subscribe to Data Sheet, Fortune’s daily newsletter on the business of technology.