To build a convincing illusion, we must first understand how reality convinces us.

For centuries, visual technologies have become better at constructing things that are not physically present.

Painting reproduced appearances. Photography captured them mechanically. Cinema constructed moving worlds. Games made those worlds interactive. Virtual reality placed the observer inside them.

Augmented reality (AR) creates a different problem. It asks something nonexistent to share perceptual space with something real.

A virtual cube must sit on a real table. A virtual instruction must align with a real machine. A surgical overlay may need to appear attached to real anatomy.

At that point, graphical realism is not enough. The virtual object must obey enough of the rules that the perceptual system uses to organise the physical world.

I came to augmented reality from a background in film, so I have always been interested in the distinction between what is physically real and what merely appears convincing. Visual effects rarely reproduce nature in every detail. They reproduce what is necessary to make the viewer believe, selecting the details that serve the scene, the story or the intended emotional response.

Computer graphics work pretty much the same way. It is not simply the technical imitation of nature, but a collection of perceptual strategies. We exaggerate contrast, simplify illumination, fake reflections and manipulate depth cues because the final judge is not physics itself, but the human perceptual system.

A small question with a difficult answer

Consider a virtual object placed on a real table through an augmented reality headset.

The object is not physically there, but it still has to appear as though it belongs to the world around it. It must seem to occupy a particular position, respond to the lighting and relate convincingly to the surface beneath it.

Most importantly, it must appear either to rest on the table or to float just above it.

Figure 1. Experimental setup for studying surface-contact perception in optical see-through AR. A participant wearing a Magic Leap 2 judges whether the virtual cube is resting on the textured physical surface or floating 6.4 mm above it.

In ordinary vision, we rely on many overlapping cues to understand spatial relationships: shadows, texture, edges, lighting, occlusion and the way objects meet one another. In augmented reality, these cues are divided between the physical world and the virtual image.

The headset can add virtual light to the scene, but it cannot fully control the light already reaching the eyes from the real environment. This makes some familiar graphical techniques, especially dark cast shadows, difficult to reproduce convincingly in optical see-through displays.

The problem is not simply how to render a virtual object.

It is how to make the object participate in a world that the computer does not completely control.

And participation always involves selection. The system cannot reproduce every physical detail of the environment, nor does it need to. It must decide which visual information to add, which information to preserve and which information can be safely ignored in order to communicate the intended spatial relationship.

When realism becomes a problem

It might seem that the solution is straightforward: make the virtual object as solid and visually convincing as possible.

But a virtual object is not perceived in isolation. Its appearance has to be interpreted together with the real surface, the surrounding texture and the available lighting information.

If the object becomes too transparent, it may lose visual presence. If it becomes completely opaque, it may hide information that helps the observer understand its relationship with the environment.

This creates a tension between seeing the object and locating it.

A rendering can make an object look more substantial while making its position harder to judge. Conversely, a small amount of transparency can preserve information about the real scene that helps the visual system decide whether the object is touching the surface or is separated from it.

In my research on optical see-through augmented reality, this balance appears most useful at 60 to 80 percent opacity levels rather than at either extreme. The result was not a universal setting that solves the problem for every scene, but evidence that perceptual clarity does not always increase with visual solidity.

The most convincing object is not necessarily the easiest object to place in space.

This is not a failure of the medium. It is a reminder that every medium has to choose what kind of information it is trying to communicate. If the purpose is to make an object visually vivid, opacity may help. If the purpose is to communicate contact with a real surface, preserving some information about that surface may be more important.

Confidence is not the same as perception

This distinction also appears in the way people evaluate augmented reality.

A virtual object may look sharp, solid and realistic, leading users to feel confident that it is correctly positioned. Yet confidence does not always correspond to perceptual accuracy.

The rendering that feels most convincing may not provide the information needed for a particular judgement.

This matters because AR systems are often assessed through questions such as:

Does the object look realistic? Does it feel well placed? Does the rendering look natural?

These are important questions, but they describe subjective experience rather than perceptual performance. A user can feel certain and still misjudge whether an object is touching a surface.

A convincing simulation can therefore influence not only what we see, but also how certain we feel about what we see.

It can also shape what we attend to. By increasing contrast, opacity or visual salience, a system may direct the user toward the information the designer considers important. This is one of the fundamental functions of media: not to present everything equally, but to organise perception around a particular purpose.

From the lab to the real world

The experiment was deliberately controlled. Background textures, object appearance and environmental illumination were varied systematically so that specific perceptual relationships could be isolated. Real environments are much less cooperative.

Figure 2. From controlled laboratory conditions to uncontrolled real-world environments. The next challenge is to determine whether the same perceptual relationships hold across natural textures, changing illumination and visually complex scenes.

Moving beyond the lab therefore means asking whether the cues that work under carefully controlled conditions remain useful when surfaces, shadows, lighting and visual clutter change continuously. That transition is not simply a matter of adding realism; it tests whether the perceptual mechanisms identified in the laboratory are robust enough to guide AR design in everyday environments.

Find out more about my research project here.

References

[1] J. Ping, B. H. Thomas, J. Baumeister, J. Guo, D. Weng, and Y. Liu, “Effects of shading model and opacity on depth perception in optical see-through augmented reality,” Journal of the Society for Information Display, vol. 28, no. 11, pp. 892–904, 2020, doi: 10.1002/jsid.947.

[2] A. Erickson, K. Kim, G. Bruder, and G. F. Welch, “Exploring the limitations of environment lighting on optical see-through head-mounted displays,” in Proceedings of the 2020 ACM Symposium on Spatial User Interaction (SUI ’20), 2020, Art. no. 9, pp. 1–8, doi: 10.1145/3385959.3418445.

[3] S. Ikeda, Y. Kimura, S. Manabe, A. Kimura, and F. Shibata, “Shadow induction on optical see-through head-mounted displays,” Computers & Graphics, vol. 91, pp. 141–152, 2020, doi: 10.1016/j.cag.2020.07.003.

[4] D. Kersten and P. Mamassian, “Cast shadow illusions,” in The Oxford Compendium of Visual Illusions, A. G. Shapiro and D. Todorović, Eds. Oxford, U.K.: Oxford University Press, 2017, pp. 214–220, doi: 10.1093/acprof:oso/9780199794607.003.0020.

[5] M. Ni, H.-T. Chen, G. Coppin, and É. Peillard, “How opacity and background affect surface contact perception in optical see-through AR,” HAL Open Science, hal-05716405, 2026. [Online]. Available: https://hal.science/hal-05716405