The Wait is a mixed-reality experience developed for Hololens. It was created by a five-person team in two weeks as part of the Building Virtual Worlds course at Carnegie Mellon University’s Entertainment Technology Center. Players are given the role of an anxious mother as she waits in a hospital room while her daughter is receiving surgery. The player may interact with objects in the room that recreate memories in the form of holograms. After the mother engages with her memories, the doctor returns to report on whether the surgery was successful. Here I will discuss the challenges we faced and the design decisions we made while rapidly developing a mixed-reality experience using Unity 5.
Challenge: Mapping virtual objects to a real world environment
Early on in our development process, we knew that we wanted players to interact with real-world objects to trigger events in the virtual space and for the virtual characters to interact with the real world objects without seeming out of place. In order to accomplish this, we measured the objects and the distance between them. We created 1:1 scale models to serve as placeholders in the Unity scene. These models also used a custom shader to create proper clipping when a real object was in between the player and a hologram.
To make calibrating the objects easier, we set a real world object’s virtual counterpart at the Unity scene’s origin position (Vector3(0,0,0)). For example, one of our objects was a hospital bed, so we set one of its legs at the origin. When we were ready to deploy and test our builds, we aimed a virtual cursor at the bed’s leg and used custom voice commands to display the placeholders and calibrate their position based off of the real world bed. After, while wearing the Hololens, one of us moved the real world objects to match the positions of their virtual versions. Occasionally, calibrating in this fashion did not yield perfect results, so we added keyboard inputs that could slightly shift the objects in a chosen direction. The result of such careful calibration allowed our holographic characters to exist more authentically within the real world space.
Challenge: Giving the player engaging interactions between physical and virtual objects
Initially, we had player interact with real-world objects using Hololens’ gaze and gestures. Players would stand at a distance from the objects and pinch at them to start the memories. We felt that players were not actively exploring the space and looking into the meaning behind the objects, reducing their immersion. Thus, we sought a way to track the player’s hand, so that they could touch a physical object in order to trigger events within Unity.
Utilizing Microsoft’s open source HoloToolKit for Unity and a script by Microsoft software engineer Ritchie Lozada, we were able to implement a way of tracking a player’s hand and assigning it a position in Unity, yet not without limitations. In order for the hand tracking to work, the Hololens must first detect the player’s hand in the “ready” position (the index finger raised with the remaining fingers closed). Unfortunately, the guest can’t simply touch objects, but must first put their hand in the ready position, then touch the object. While this still may not be the most natural interaction, the physical touching caused players to move around, examine the objects more closely, and develop a stronger understanding of the object’s relation to the story.
Early on, one of the major points of feedback we received from playtesters was that they felt little to no connection with their virtual daughter: Alice. This prevented them from becoming immersed in the experience. Along with a few script and story changes, we decided we needed to have meaningful interactions between the player and the virtual Alice in order to strengthen their emotional connection. Thus, we created three physical interactions with the Alice hologram. Despite her virtual nature, Alice’s dialogue and animations caused players to perceive her intractability. They are prompted to hold her hand, hug her, and bring her a physical piece of paper so she can draw them a picture. Through performing these actions, players became more empathetic towards Alice which increased their overall engagement.
Challenge: Avoiding the limitations of the Hololens’ viewport
One of the current technological limitations of the Hololens is that its viewport only takes up a relatively small portion of a user’s visual field. This was initially a major issue for us, as we needed the player to see and interact with life-sized holograms. So, we made a few changes in order to orient the player in the proper position to view the holograms. First, we reduced the size of the daughter character, allowing her to fit inside the viewport at most distances. Second, since the character holograms were in close proximity to the bed, we increased the distance between the bed and the other interactable objects. Thus, the player would generally be at a good viewing distance right after they touched an object. Finally, we added a particle/trail effect after the player touches an object. These trails guided the player’s eye to where the next memory would begin, thus they would naturally orient themselves to best see the holograms.
Unfortunately, there were still some limitations of the viewport that we were unable to overcome given the overall design of the game. When a player is prompted to move closer to Alice, it would often mean that she would be clipped by the viewport, often breaking the player’s immersion.
Based on my experiences with the aforementioned challenges and the solutions my team and I came up with, I conclude that the Hololens is effective at delivering immersive experiences through the following ways:
- Give the player meaningful interactions with both physical and virtual objects.
- Have the player deliver an object or make conversation to characters. Cause them to interact with a physical object that then changes the virtual space.
- Have the virtual objects interact with physical objects.
- Make a virtual character sit on a physical chair, examine real objects, or approach the player to converse.
- Use indirect control to position the player to properly see what you want them to.
- Implement spatial sound to cause the player to turn in a specific direction. Control the distance between where the player is likely to stand and where holograms appear so that they can be seen fully. Use particle effects or other visual cues to guide the player’s vision towards holograms.
- Understand the tradeoff between player immersion and emotional connectedness to the virtual world.
- Allowing players to interact closely with holograms can increase their emotional connection to them. However, the player’s sense of presence could be lost if the holograms become clipped by the Hololens’ small viewport.
If these points are properly implemented, I believe that the player will develop an understanding of the connection between the physical and virtual worlds, thus leading to greater immersion and emotional engagement.