Pop-up story book in virtual reality

Creating a guided exploration of virtual worlds using established patterns in storytelling and game design

Roles

• Interaction Design

This work is from my role as an experience designer at an advertising agency

As part of a marketing campaign, a wine company commissioned artists to create illustrations imagining the world around the brand’s mascot stag. These illustrations were used in promotional videos that show the mascot leaping out of the bottle and running through the scenes. My role in this project was to concept and define a mobile AR or VR experience as an extension to these ads.

Rather than create a literal translation of the short videos in VR, I adapted the concept into an engaging story that works like a digital pop-up book. My understanding of game design, animation, and the VR medium were crucial to the success of this project.

Methods

Aligning on an interaction pattern

The core concept of the running mascot can be adapted into AR/VR in a myriad of approaches. Since this was a project with many stakeholders and team members across disciplines, it was important to align on the direction before getting too far in the design process.

Mixed reality is still a relatively niche medium, so to help define the idea I presented rough mockups of several interaction patterns, each with their own pros and cons:

In the end, the panorama option was selected as it aligned most closely with the sense of openness and freedom that was integral to the campaign’s creative direction.

Balancing freedom and guidance

In a 360˚ video, the sense of immersion and freedom gained can also be overwhelming. This is especially an issue on mobile devices because of the limited field of view, inaccurate world tracking, and lack of spatial audio cues – all factors that make it harder for users to orient themselves compared to immersive experiences on VR headsets.

To guide the user in exploring the panoramic worlds, I used the mascot as the focal point around which all the action unfolds. The mascot would run through worlds that wrap around the user, who can follow along with the camera. The user is still free to look around the world on their own, but an indicator will point back towards the mascot when it’s offscreen, nudging the user back to the center of the action.

This is an established user interface pattern in video games, especially open-world games where waypoint indicators guide players to a direction while still allowing them to explore at their own pace.

Camera framing

To keep users engaged throughout the experience, I decided on a structure where the scenes evolve over time, as opposed to animated but looping dioramas that can feel soulless. A changing world creates a continuous sense of progression and encourages users to keep going. However, using this approach in a VR experience presented some unique challenges.

In conventional films, filmmakers can frame the viewers’ attention shot after shot to build a narrative. In panoramic experiences where users can choose their own perspective at any moment, special considerations are needed to ensure one-time events do not occur offscreen. To direct users to look in the right direction, I adopted an interaction pattern where each animation is triggered at a checkpoint that requires user action.

The mascot indicator and interaction checkpoints work together to nudge users into specific vantage points, effectively framing the shots like in a film. The mascot leads the user to each checkpoint and waits for them to press a button, which triggers an animation in the scene. The buttons are strategically placed where the subsequent animation would unfold, implicitly forcing the user to look at the important part of the scene.

Narrative structure

With the mechanics of the experience defined, I identified parts of the illustrations that could be rearranged and animated to create a loose narrative. We were limited to using the existing static illustrations, so I looked for elements that could be animated using only 3D translations or rotations. Although the animations were simple, they created an effect delightfully similar to pulling a tab in a pop-up book.

In selecting the interaction points, I looked for patterns that can be repeated to create a narrative structure. The animations I proposed at each checkpoint follow a theme of encounters with other creatures – including a leaping whale, a fanning peacock, and a roaring bear. I placed an otherworldly bird at the end of each scene as a consistent motif for transporting the mascot to the next world. To culminate the story in a dramatic finish, I arranged the scenes to end at an encounter with a creature similar to the mascot.

Even though users don’t have real choices that would change the outcome of the story, the simple interactions give them a sense of agency as though they are helping the mascot. With more budget, this interaction pattern can easily scale up to multiple paths that depend on their choice at each checkpoint. This would elevate this on-rails experience to something more complex and replayable, like a choose-you-own-adventure story.

Annotations

I created wireframes for each screen in the experience. These served as a storyboard cataloguing all the parts of the UI for the visual designer to style. The documentation also included functional annotations detailing the user flow and interactions, which provided clear guidance for the developers to implement the experience.

Results

Accolades

• Webby Awards 2022 – Honoree in Advertising, Media & PR, Augmented Reality
• 14th Shorty Awards – Audience Honor in Wine, Beer & Spirits, Augmented Reality, Immersive