Augmented Reality—What Reality Can We Learn From It?

Cherry Thian, Asian Civilisations Museum, Singapore

www.acm.org.sg

Abstract

In June 2011, the Asian Civilisations Museum launched an integrated iPhone app—the world’s first museum app to combine augmented reality (AR), location-based gaming, and interactive features—in its special exhibition Terracotta Warriors: The First Emperor and His Legacy. Specially designed to tell a story that complements the content and flow of the exhibition in seven interactive chapters, each chapter highlights at least one object in the exhibition and attempts to engage visitors on a more personal level. As the story progresses, visitors respond to image markers that launch augmented reality and gaming experiences that breathe life into the objects.

The launch of this app was a pilot exercise, and it presented us with an opportunity to study the adoption of technology by our visitors. The integration of several interactive features in the app also allowed us to look at behaviour and interactions of users. This paper surveys the effectiveness of this implementation, explores the wider issues raised by the findings, and relates lessons learned from unforeseen challenges along the way.

Keywords: augmented reality, Asian Civilisations Museum, mobile, interactive, gaming, edutainment

1.   Introduction

The role of the museum is no longer just to exhibit significant and beautiful objects; now it must also create rich, fun, and engaging experiences for visitors. One way to achieve this is through the use of new technology. This is especially relevant when trying to reach the younger generation. Many museums implement new technologies as part of their continuous pursuit of new ways to stimulate young, tech-saavy audiences. When well designed and executed, these can add layers to objects by creating new narratives that provide alternate modes of access. Augmented reality (AR) has been made available on mobile phones in recent years, and museums have been in the forefront of experimentation to test its educational value.

The impact of AR technology is still not fully understood. While it appears to be noble and exciting to many, relatively few publications discuss its use in terms of location-based visitor experiences, or the social and educational interaction that is shaped by its characteristics and objects involved.

The Terracotta Warriors app was one of the top five downloaded iPad apps for June 2011. Visitors generally responded enthusiastically, and the app garnered much positive publicity from both local and international media. Our experience was positive overall, but there were problems, and we were presented with some dilemmas. When museums turn to interactive technologies in pursuit of new ways of engaging young audiences, where do we draw the line to avoid turning the museum into a theme park? How can we most effectively get everyone up to speed with the latest technology? As the museum works to engage younger visitors with appealing technologies, do we risk distracting them from meaningful engagement with real objects? The implementation of this app as a pilot exercise has given the museum an opportunity to investigate user interaction in situ and to gain richer insights into the factors that influence visitor response, learning assimilation, and emotional engagement. Some modifications had to be made during the course of the exhibition as the museum learned and reacted to unforeseen challenges.

2.   The study

Location: Asian Civilisations Museum, Singapore

Located by the historic Singapore River, the Asian Civilisations Museum (ACM) traces the diverse cultural roots of modern Singaporeans. The museum features collections from Southeast Asia, South Asia, China, and the Islamic world of West Asia. Within ten themed galleries, the museum charts a bold and new approach in engaging museum visitors through the displays, presentation of information, and creative use of interactive technology to weave integrated story lines for the visitor. It also presents a programme of changing special exhibitions throughout the year.

The special exhibition

In June 2011, the museum mounted the special exhibition Terracotta Warriors: The First Emperor and His Legacy in partnership with the Shaanxi Provincial Cultural Relics Bureau and the Shaanxi Cultural Heritage Promotion Centre, People’s Republic of China. This exhibition featured one hundred objects from Shaanxi province in China. Besides ten of the famous terracotta figures, the show contained important bronzes and jades from the Qin state before the time of the First Emperor. The Han dynasty inherited the legacy of unified China, and striking terracotta figures from this later period were also included to show how burial practice changed after the time of the First Emperor.

The objects were arranged in chronological order: the pre-Qin period (the Qin state before the time of the First Emperor); the Qin state, highlighted by the warriors, chariot fittings, weapons, and other tomb objects; and the post-Qin period (Han dynasty). The First Emperor united China for the first time, abolished its feudal past, unified a single written language, built an early version of the Great Wall, and directed the creation of the world’s biggest and most extravagant resting place. While the story is rich and fascinating, and the warriors are well known, we hoped to provide younger visitors a way to connect with the story of the emperor in ways other than by reading object labels and descriptions on exhibition text panels.

The ACM: Terracotta Warriors iPhone app

Design rationale

The goals in terms of learning outcomes were to promote closer interaction between visitors and objects, and a deeper understanding of the objects and the history of Qin dynasty. Several factors gave rise to the implementation of the app. First, we hoped to attract young visitors aged 15 to 30 years old and to change their perception of the museum. Second, the lack of space in the exhibition gallery for interactive installations, text, and visuals forced us to get more creative with interpretive materials. Third, we had been observing more visitors using their mobile phones to take photographs and connect to their peers online while in the galleries, and we wanted to try to incorporate that into an exhibition experience. Thus, the museum chose to explore mobile technology to provide a novel multi-sensory experience that could engage visitors on a more personal level through a story, specifically designed and scripted for the target audience.

The museum also took this opportunity to experiment with designing an intervention to try to elicit emotion that would facilitate perception. Perception is seen as “essential to aesthetic experience and leads to psychological growth and learning” (Dewey, 1934: 177). The experience was designed to echo Dewey’s principles: to allow perception to occur when the user uses the app, which would impose surprising qualities, which in turn would create new insights. Therefore, by creating an interactive experience in which participants are aware and engaged emotionally with the objects, we hope to stimulate reflection and perception through action and interaction.

To make it easier for visitors to locate the markers and have close interaction with the objects, the markers had to be placed as close as possible to the object and the label text.  One of the major considerations was making the marker attractive and as unobtrusive as possible in the gallery. Normal quick response (QR) codes did not seem appropriate for an exhibition space. We chose instead to design the markers as Chinese words written in seal script; the words also described each corresponding object.  

Figure 1: Marker for the Ancestral Bell, written in Chinese seal script for bell

Experience the Warrior Trail

Storytelling is at the heart of what museums do. Stories help to make connections with audiences and breathe life into the museum collections, capturing and eliciting a range of emotions. Storytelling  is an effective way to promote object-based learning. (Cain and Cain, 1994)

The game was designed as an interactive story in seven chapters to complement the flow and content of the exhibition, highlighting at least one object in each chapter. The story is unveiled through a dialogue between Li Si, the prime minister and architect of the Qin Empire, and his son before they were going to be executed. Through the dialogue, visitors learn about the emperor’s ambitions and hidden agendas. As the story progresses, visitors are prompted to launch augmented reality and gaming experiences that animated objects or illustrate background information. Each chapter is different, and thus the experience is surprising and fun (table 1). It takes about twenty minutes to complete the trail.

The trail begins with a short introduction.

Figure 2: The game begins with a display of seven locked chapters, each marked by the image of an object that corresponds to one of the markers. The Ancestral Bell was the first object connected with the app.

Chapter	Experience
Ancestral Bell Pre-Qin period	AR bell that allows user to ring and hear the actual sound.
Battle Tactics	Learn how the Qin army was commanded on the battlefield. Pound the drum to advance the troops, and ring the bell to signal retreat (with sound effects).
Imperial Edict Qin unified China	Learn to write “Emperor” with correct stroke order in Chinese seal script—the first standardized writing script in China.
The Tomb	An animation shows the emperor’s tomb construction based on archeological records.
Painted Army	AR coloured warrior—to show how the terracotta warriors might have originally looked (before disintegration of the paint used to decorate them).
Invincible Archers	An animation to show how the crossbow mechanism works, followed by AR warriors in alternating firing ranks in battle.
Downfall The finale—the fall of Qin dynasty	A giant AR tomb raider breaks through the wall, intending to destroy the tomb treasures.

Table 1: Interactive features by chapter

Figure 3: Instructions on where to locate the markers and how to activate them

Illustrations are shown in the background as Li Si’s dialogue with his son progresses. The objects were chosen based on the significance to the exhibition story line. Visuals are powerful in giving the context of how the object was used as the story progresses (Cain and Cain, 1994).  

Figure 4: Charioteer in action. This visual is shown at the chariot marker, which launches a battle-tactics game when activated.

Pre-visit experience

Two AR markers were designed for publicity and outreach purposes: a giant marker that spawns a three-dimensional (3D), animated, life-size warrior and a mini marker that activates an AR curious warrior who walks where the user taps. The giant markers were placed at a rapid transit station, libraries, and in the lobby of the museum. The mini one was used on printed materials such as newspapers, brochures, and flyers. These markers were also made available on the ACM website. Once the app is downloaded, users can start using the two markers and reading the prologue of the story. However, they cannot progress if the chapters are not “unlocked,” which they can only do at the exhibition gallery. 

Figure 5: Markers for publicity

Technology

The app used a technology called String AR to superimpose animated 3D objects into the real exhibition gallery space. The technology was not commercially released at the time of launching. Displaying high-quality AR content (such as a life-sized 3D terracotta warrior surrounded by fire, or a miniature warrior marching across a newspaper or flyer) requires high frame rates and extremely responsive tracking in varying light levels—features currently provided only by String AR. The interactive story and mini-game features were made with Unity3D, which is one of the world’s leading engines for creating 3D video games and real-time 3D animations on a smart phone or a web-browser.

The huge challenge for the app developer was the integration of all these features into one seamless exhibition tour application. This ambitious combination of interactive narrative, augmented reality, and mini-game content truly has never been attempted before and raised some serious technical hurdles that were a major challenge to overcome.

Study approach and data collection

A total of 140 user observations and surveys were conducted over the course of three months, including weekends, targeting visitors aged 15 to 30 years old (table 2). Gallery assistants walked around and stood close to exhibits with markers, taking notes of interactions and occasional visitor conversations that could be overheard. In addition to written notes, photographs were taken of social interactions and behaviours as documentary data for further analysis. Some users were shadowed by researchers and were interviewed and asked to freely comment on their preferences and feedbacks. The visits varied between thirty minutes and two hours.

The following age groups were observed and/or interviewed:

Age	Number of participants
15–22	72
23–30	41
31–35	20
36 and above	7
Total	140

 

Table 2: Distribution of participant’s age groups

Findings and observations

The app appealed to visitors between 15 and 22 years old. A sharp decline in interest was observed for visitors aged 30 and above. This was also supported by the completion rate of the trail by each age group (see table 5 below).

Observation and photographs revealed patterns of behaviour and interactions associated with various age groups.

Downloading

Wireless connection and phone reception is very weak inside the museum building, and especially in the special exhibition gallery. Before the exhibition was launched, several routers were installed to boost the connectivity strength. However, due to the massive stone construction of the building and network infrastructure in the area, only limited bandwidth could be made available. Network congestion occurred when the crowd started to download the app concurrently at the first and forth marker. This behaviour will be discussed further in the following section.

Visitor engagement with the app

Discovering and learning to use the app

Augmented reality is still a relatively novel idea and is poorly understood by the general public. In fact, only 24 out of 140 participants had heard of the AR technology before the exhibition. Out of these 24, only 2 had tried a somewhat similar experience before. Despite the presence of signs to instruct visitors how to download the app, not many iPhone users downloaded the app immediately upon entering the special exhibition gallery. But once they saw other visitors using and talking about the app at the first marker, new visitors—both iPhone and non-iPhone users—quickly became interested. This supports the “honey pot effect” described by Brignull, in which visitors are attracted by the interest of others (Brignull and Rogers, 2003). This chain of interaction continued for up to ten minutes. When an active user was present to activate the marker with his or her phone, other people would stop and observe, and some would attempt to download the app (with or without an iPhone) and learn how to use it from observing others. The learning pattern was interesting: many instinctively launched the camera programme on their phones (both iPhone and non-iPhone) and took a photograph of the marker. This denotes familiarity with QR technology. A handful of them just walked away after taking the photograph and proceeded to the next one to capture the image of another marker, but some asked questions of those who managed to use the app successfully.

When respondents were asked if they needed assistance with playing the game, the majority declined, and instead chose to work out the app by trial and error. (table 3)

 

Age	Trial and error	Ask another person
15–22 N=72	15 (21%)	—
23–30 N=41	23 (56%)	8
31–35 N=20	14 (70%)	4
36 and above N=7	4 (57%)	1

Table 3: Methods of overcoming challenges by age groups

It was interesting to find out that 21 percent of the 15-to-22 age group preferred to solve problems by trial and error rather than by approaching others for help. The other 79 percent did not seem to encounter any problems during the game play. It was also noted that during the chain of interaction, when visitors observed others using or asking about the app, many visitors became very excited and started downloading the app at the first marker. This resulted in network congestion, which caused the download speed to be extremely slow. On average, the waiting time could be up to three minutes (the size of the app is twenty megabytes). This created problems. Two observations were made: a handful of visitors gave up downloading the app; the other group wandered off to look at other exhibits while checking on the download status. This issue is critical and can determine the success or failure of an app (Allen, 2004). The app must be made available so that visitors can experience success early and feel competent. Visitors will likely move on and give up if the frustration builds up at the very beginning of the journey (Ciolfi and Bannon, 2003).

After the first marker, most users had become competent and confidently activated the second marker. The majority in the target group, aged 15 to 22 years old, was observed to rush to complete the trail. Here are some of their responses:

“I just can’t wait to see what the next one (marker) is!”

“I want to finish the story first, and then I will look at the exhibits again.”

“This is so cool! I want to complete this game first.”

Educating users about AR

Explaining the concept to the visitors was challenging. The only way to convey the idea was through demonstration (Ormrod, 2003). Three weeks into observing this behavior (highlighted above), the museum installed an AR photo opportunity booth, allowing visitors to take their photo with an AR warrior. that the booth offered infographics and a download tutorial and user’s manual at the main entrance (where visitors queued to purchase entry tickets). A constant crowd was seen downloading the app near the ticket counter, and the crowd grew each time new visitors saw others at the station. This helped to ease the network congestion by half in the exhibition gallery.  And when they arrived at the special exhibition gallery, visitors who had downloaded or learned about the app in the main lobby were excited and more ready and able to experience the full story trail.

Figure 6: AR photo booth at museum lobby

Social interaction

Besides the educational aspect of the app, we were also aiming to promote social interaction and discussion in the exhibition and online. We compared different types of conversations and interactions around exhibits with the markers and at other exhibits without markers. Observations have shown that parents with younger children (aged 12 and below) used their iPhones or iPads to guide their children’s thinking and attention, drawing connections between the objects and the interactive components. Some were observed reading the story to their children. After the first marker, the children were often seen leading their parents to look for the second marker. For families with older parents, the children were the ones who taught their parents how it worked.

A different kind of interaction was observed at certain AR markers (such as the painted army, invincible archers, and downfall). Researcher observed that visitors were involved in role-playing with the AR warriors, and they established connections and bridged the gap between the two realities. Pictures were instantly posted and shared with friends on Facebook. This was more common among visitors who came in groups.

Figure 7: Users’ engagement with AR experience

Reading the instructions

Another critical observation was that 40 percent of the users did not read the instructions on the screen of the first marker, the ancestral bell. Part of the activity there is ringing the bell. Users were surprised when the researcher asked if they knew about swiping the screen to ring and hear the actual sound of the bell. Here are some responses given:

“I was so intrigued by the appearance of the bell that I did not notice the instructions above.”

“How do I ring the bell?”

“Wow, you mean I can ring the bell on my screen?”

Figure 8: The AR ancestral bell

Similarly, many younger users skipped the instructions on the battle tactics game and started to play the game straightaway. They were observed figuring out on their own what the drum and bell were for, as well as the goal of the game during the game play.

Figure 9: The Battle Tactics

Game popularity ranking

The battle tactics chapter was highly favoured among the 15 to 22 year olds, which was a surprising finding.(table 4)  Respondents in this group explained that they liked the idea of playing a game in the gallery, and they also enjoyed finding out about the use of drum and bell in war.

The AR features were ranked after the battle tactics game. The ancestral bell was the least popular AR feature in the trail, mainly because it was the first object with a marker and many were still trying to learn how to use the app at that point. And as noted above, a lot of visitors did not know that the bell could be rung.

Some users commented that the animation of the tomb was less interesting and interactive compared to the other features. They preferred activities that they could engage in and that had a goal to pursue—a gaming aspect.

Age	Ancestral bell (AR)	Battle tactics	Imperial edict	The tomb	Painted army (AR)	Invincible archers (AR)	Downfall (AR)
15–22	5	23	4	3	9	21	22
23–30	2	16	4	1	5	4	11
31–35	4	5	4	0	8	4	6
36 and above	0	0	1	0	1	0	0
Total	11	44	13	4	23	29	39

Table 4: Popularity ranking

Completion of the game

Out of 140 respondents, 108 completed the game (table 5). The breakdown of the figures corresponding to age groups is as follows:

Age	Did not complete the game because:				Percentage of users did not complete the game
	Duration of the game is too long	Instruction is unclear	Technical difficulty	Rushing for time
15–22 N=72	1	1		1	4.2%
23–30 N=41	7		3	1	26.8%
31–35 N=20	10	2	1		65%
36 and above N=7	3	1	1		57%
Total	24	3	5	2

Table 5: Number of users that did not complete the game, and their reasons

The level of interest dropped significantly for users over 30 years old. Most thought that the story was too long, and they lost interest after a while. Those who completed the trail were also observed to be engaged more deeply and for a longer time with the contents. Teenagers compared results and viewed content on shared phones, and engaged in playful treasure hunting as they raced to complete the story trail.

Some users were also spotted tapping frantically on the screen to skip the story to activate the next marker. When they were asked why, the general response was that the story was too long. They grew impatient going from dialogue to dialogue. Out of 140 respondents, 78 thought the game play was just fine, while 33 thought the dialogues were too long. 

Measuring learning outcomes

The two main learning outcomes that we set out to achieve with the app were learning about the history of the Qin dynasty and engagement between the visitor and the object on display. When queried, 112 users responded that the app allowed them to learn about the rise and fall of the Qin Dynasty. Additionally, 124 responded that the app motivated them to examine the exhibits with markers more closely, which allowed them to better understand them in context. Some responses recorded were as follows:

“It is interesting to see how they used the drum and bell in the war. It is quite different than the Lord of Rings.”

“The animation that shows how crossbow works is interesting. I can’t tell what that metal was for initially.”

“The coloured warrior is cool! I always thought they looked like that (without colours).”

“The story is interesting though it is a little too wordy. But I guess I can still read it after the exhibition.” 

Changing of perception

One of the key challenges faced by the museum is to attract the younger generation to exhibitions and programmes. It is critical to repackage the experience so as to change the mindset—to make them think about museums in a more positive way.

“Museum is no longer boring”

“It’s amazing how technology can enhance a museum visit. Static displays are no longer cool!”

“Very creative and educational, fully integrated with the exhibits to bring out the best of history!

“Was blown away at the interactive features of the app and its integration with the exhibition. Made the visit totally cool, and my friends all agreed!”

“Keep it up acm. Love those story, 3D effect and game..thanks”

“This is one amazing app. It makes the whole Terracotta exhibition come alive and immensely interesting. Get it now.” 

Post visit experience

Most of the comments regarding the post-visit experiences were gathered through blogs, Facebook, and from returning visitors. The markers for publicity were exceptionally popular. Users printed out the markers and used them at home or at work. Pictures taken with the AR warriors became viral on social media, drawing much attention and free publicity for the exhibition. Visitors did continue to play with the app after they left the museum. They were able to share their post-visit experience with others who could not come to the gallery. Some commented that they continued to read the story in the app after they left the gallery.

3.   Discussion and conclusion

It is always a challenge to strike a balance between entertainment and education, mainly dependent on a visitor’s emotional disposition and motivation. With the implementation of the app, it was noted that users generally spent a longer time in the museum, and they stated that their visits had been improved through the use of the app. The app encouraged and enabled the kinds of interactions that initiate sense-making, construction of personal and social meaning, dialogue, and emotional responses. It is still unclear how much users actually learned about the objects, but stirring the interest and motivation to step into the gallery is the most critical factor. Once that is achieved, at least the museum is given a chance to show what it can offer.

Though visitors are not, of course, allowed to touch the objects, we hoped that the app would add other sensory elements to the visual experience of looking, allowing the visitors to more readily imagine, intuit, and remember. All these intersecting stimuli across sensorial interactions inspire greater, more meaningful experiences with objects (Dudley, 2010).

This study also told us about visitors’ readiness and motivation to adopt new technology. While the digital natives navigated the trail intuitively, some users appear to have been put off by the technology. But this experiment gave us an insight into how the system could be improved to reduce learning curves across target groups.

Undeniably, the mobile technology offers layers of information that otherwise could not be displayed due to the aesthetic and space concerns. However, the next challenge is to retain and expand attention spans and promote learning through multiple layers and meaningful activities (Ciolfi and Bannon, 2003). It was also noted that visual is better than textual information. Some visitors became bored with the long narrative segments. The storytelling component could be better enhanced or woven into the interactive features, making it a reward for action and completion of goals.

ACM also plans to look at an app that can be used on multiple platforms: Many Android users were disappointed that they could not participate. We hope to cater to smart-phone users in general, rather than only to iPhone users, with the next application. One consideration is to use open-source platforms in the future, when the technology is available.

Moving forward, perhaps museums have to consider emotional aspects in designing “edutainment” experiences. As Phelps pointed out, emotion facilitates early visual processing, facilitates perception, and potentiates attention (Phelps, Ling, and Carrasco, 2006) and may help to produce persistence and direction of behaviour after a reinforcing stimulus has occurred to help achieve a goal or goals (Rolls, 2007).

4.   Acknowledgements

Thanks to Richard Lingner for providing valuable inputs to this paper; Ministry of Information, Communications and the Arts for funding this project; Magma Studio for developing this app; Alan Chong, T.Sundraraj, Karen Chin for supporting this research and implementation; and to the rest of the exhibition team for making this exhibition memorable and a success.

5.   References

Allen, S. (2004). “Designs for Learning: Studying Science.” Science Education, 88(S1): 17–33.

Brignull, H., and Y. Rogers. (2003). “Enticing people to interact with large public displays.” Proc. of INTERACT'03: 17–24.

Cain, N., and G. Cain. (1994). Making Connections: Teaching and the Human Brain. New York: Addison Wesley Longman, Inc.

Ciolfi, L., and L. Bannon. (2003). “Learning from Museum Visits: Shaping Design Sensitivities.” Proceedings of HCI International 2003.

>Dewey, J. (1934). Human nature and conduct. New York: Random House.

Dudley, S. (2010). “Objects, sense and feeling.” In Museum materialities (p. 8). New York: Routledge.

Ormrod, J.E. (2003). Human Learning, Third Edition. Upper Saddle River, NJ: Merrill/Prentice Hall.

Phelps, E.A., S. Ling, and M. Carrasco. (2006). “Emotion Facilitates Perception and Potentiates the Perceptual Benefits of Attention.” National Institutes of Health: 292–299.

Rolls, T. (2007). A Neurobiological Approach to Emotional Intelligence: 72–100.