Fragments is a mystery-solving game that unfolds in the actual, physical space around a player (Image: Asobo Studio).
The future of immersive gaming experiences
Fragments is a crime thriller augmented reality (AR) game for the Microsoft Hololens. Produced by Asobo Studio, it serves as a fantastic example of how gaming and storytelling narratives can be enhanced by augmented reality. I approached the game from a user experience perspective to analyze how it compares to traditional console gaming, the areas where it succeeds, and what we can learn from it to improve future AR experiences.
TL;DR
– Gaming in augmented reality creates rich and complex experiences by layering digital content on top of the player’s own environment.
– Learning from what Fragments does well can help us to define key UX principles for augmented reality experiences, such as giving the user autonomy in space and guiding them with help text and audio cues.
– Fragments’ immersive storytelling and interactive characters create a compelling narrative that helps the player be more emotionally invested.
– The rise of mobile AR experiences and voice control mean that the future of AR gaming is not far on the horizon, and Fragments is one of the best examples of where this technology can take us.
Gaming on screens vs. gaming in AR
In traditional 3D console games, a player has either a first-person or third-person view of the scene, and can use motion and camera controls to move the main character through the game. In augmented reality, the player is the main character. This means that the game plays out from the player’s own perspective, making them more emotionally invested in the outcome of the story as it unfolds around them. Every action in Fragments uses a combination of the player’s gaze as a cursor and either voice or air-tap for selection. The shift from console to AR thus simplifies an important key to gaming: replacing the controller with (mostly) natural human gestures.
Another key difference is the game’s environment. In both console and virtual reality (VR) games, the player is transported to another world. Environments can be boundless, complex, and fantastical. In AR, the game takes place in the player’s own home or activity space, with scenes and assets layered onto the environment around them. This creates an experience that feels infinitely more personal and interactive, with characters and clues joining the player in a pre-existing bounded space.
This means that each time the player maps a new space, Fragments looks completely different. Though the storyline remains the same, the location of characters, evidence, and minigames changes based on the environment, providing a high level of replay value to the game.
,
,
The story of Fragments unfolds around the player in their own space in mixed reality.
This immersive space adds a whole new layer of complexity for gamers, making the user experience more immersive and creating a layered reality on top of their own. After playing Fragments in my living room, I realized I saw my home in a whole new way. The space in which I spend most of my time became a digital canvas covered in hidden objects and puzzles to solve.
AR UX Principles we can learn from Fragments
Fragments provides one of the most delightful user experiences for gaming in AR, and the following are UX principles we can learn from what the game does well that can be applied to other augmented reality experiences.
Physical adaptation of digital assets
User experience in augmented reality games generally follows the pattern set by console games, with some spatial adjustments. Menus in console games are displayed via a start or select button, and generally pull the player out of the experience by pausing the game. In Fragments, menus are always visibly pinned to the wall during gameplay, and are accessible at any point. This means that the player never truly “leaves” the game, and participates in a much smoother experience.
,
,
AR menus are visibly pinned to a physical space rather than being tied to a pause button.
Occlusion of digital assets behind real-life objects
Fragments makes fantastic use of the Hololens’ occlusion capabilities. Occlusion is the hiding of digital assets behind real-world objects. This is no easy feat, as it requires an incredibly accurate map of the environment before the game begins, and Fragments does an excellent job of getting this right. It makes the experience of augmented reality that much more real, forcing the player to look for clues around obstructions, beneath tables, and in crevices, much like a real-life detective.
,
,
Occlusion of digital assets behind real-world objects in Fragments
Giving the user autonomy in the space
While console games provide limited controls for how and where a player interacts with objects, AR games give players much more autonomy. Using Uncharted 4 as a comparison, it’s obvious that object interaction is significantly more structured in a console game, with limited viewing and selection capabilities. Once a player selects an object, a menu appears with a set of control options to indicate ways for the player to interact with and exit viewing the object.
,
,
Interacting with an object in Uncharted 4 produces a list of button options (Source: IGN).
In Fragments, object interaction is simpler and more natural, as it involves only gazing and air-tapping. These simplified gestures mimic the experience of lifting and analyzing objects in real life. Additionally, objects are anchored to a physical location in space in AR, so the player may step away from the object or walk through it. This means that object views require an extra layer of intelligence, and Fragments automatically exits the view mode of objects which the player walks away from. By accounting for unusual player behavior, AR games give the player more autonomy in how they navigate through the space, and thus provide for a more realistic and natural user experience.
,
,
Controlling a drone in Fragments is reminiscent of piloting aerial vehicles with a handheld controller.
Guiding the user with help text and audio cues
Because of the autonomy it gives the player in moving through the game environment, Fragments employs several strategies to orient the player and ensure that they are able to navigate the game and follow the story.
Using arrows that follow the field of view, it prompts the player to move through the space with help text such as “Look Here” and “Go Here,” even specifying if the desired location is behind a physical obstruction. The game also provides audio feedback on the majority of interactable objects, providing greater backstory and hints to keep the player on track. These features enable Fragments to give the player freedom in navigating through the story, while providing guidance to help them adapt to augmented reality.
,
,
Help text guides the player through the space.
Indicating interactions with visual feedback
Similar to traditional console games, Fragments provides visual feedback when objects are able to be interacted with. The 4-arrow cursor expands to surround an interactable object with 8 arrows, sometimes even spinning to direct attention to something that can be further analyzed. This selection technique takes advantage of the familiar interface console gamers are used to, and adds an elegant layer of animated feedback to make it more useful for the player in AR.
,
,
Arrows following the player’s gaze expand and animate to indicate interactable objects in Fragments.
Providing voice control
Though replacing a controller with the player’s body simplifies the core aspects of gameplay in Fragments, using the head as a cursor and constantly lifting an arm to air-tap can get tiresome over longer periods of time.
In this regard, voice plays a major role in the game, giving the player the option to make their way through it using vocal cues. Voice control is used extensively in Fragments, with a mic icon indicating actions that can be activated vocally appearing near almost every action. This also comes in handy for activating tools and menus used to gather clues, as navigating the space via voice is much more convenient than utilizing the menu pinned to the wall for each new action.
,
,
The mic icon appears next to any action that can be activated via voice.
Improving on the UX of Fragments
Increasing accessibility with background contrast
When an object is being viewed in Fragments, key evidence discovered in the game is highlighted in orange and identified with annotations. This is useful when drawing conclusions from clues, as when a player interacts with each piece of evidence, relevant clues they have already gathered are automatically displayed.
Though these clue annotations are no doubt effective in aiding the player through the game, they would be easier to read if they appeared in bolder text against a darker background. Additionally, selection arrows in a different color could indicate that whether evidence can be further analyzed, and more annotations could appear while the object is selected with the player’s gaze. This could be implemented for an easier and more accessible version of the game, as it simplifies reviewing evidence to reach conclusions. These changes would retain the sleek and professional UI of the game, while increasing the contrast and providing the player with a better experience.
,
,
Existing UI of clue annotations (left) vs my suggested improvements with improved accessibility (right).
Limitations of AR Technology
Some of the major pitfalls of the Hololens come from an incorrect scan of the room before the game begins, which can lead to improper placement of AR assets and thus an unsatisfying game experience. Mirrors, windows, and other non-opaque surfaces also cause glitches, sometimes leading the game to expand the space to undesirable areas.
More general disadvantages include using the head as a cursor, using air-tap as the main interaction tool, and the weight of the Hololens, which can cause a minor headache when worn for extended periods of time. As AR becomes more and more mainstream, these problems will all be solved with advances in spatial mapping, eye-tracking technology, and AR headsets.
Fragments shines with immersive storytelling
What truly sets Fragments apart from other games is its interactive characters and fantastic storytelling. From examining evidence and learning the backstory of different characters, to hearing the characters’ emotions come through in cutscenes, the narrative is compelling and thoughtful, keeping the player hooked from start to finish.
The characters’ artificial intelligence is particularly impressive. They follow the player’s eyes as they move around, and similar to objects and evidence, will adapt to the mapped environment around them. Depending on the space that has been scanned for the game, characters will stand, sit on chairs, and even lean on walls. The fact that they make themselves at home in the player’s space absolutely blew me away. This natural element makes the characters feel more real and human, and helps the player to create a deeper emotional connection to the story.
,
,
Source: Prototypr