info @ archimuse.com
published: April, 2002
Cruising Through History Wired
Patricia Gillard, User Insite Pty Ltd, Australia
This paper presents the results of an evaluation of History Wired conducted at the National Museum of American History (NMAH), Smithsonian Institution, Washington, DC, during March and April 200l prior to its launch in August, 2001. The methods were based on ethnographic and constructivist approaches rather than heuristic or usability testing. Eleven groups of visitors (31 individuals) viewed the History Wired site which was in development at the NMAH. Visitors were asked to use the site together for five to ten minutes, and then to answer some interview questions. Observation of their uses of navigation, as well as choices for more information and general movement around the site, were noted. Their comments during the session as well as the interview were taped for later analysis.
The results showed that all visitors were very engaged with the site, even if they had limited Web knowledge. The aims of designers, to give direct access to the user, and to provide many ways of finding and enjoying objects, were well realized. Visitors used the site in distinctive ways, including reading from left to right, following subject interests and themes by use of the keywords, and navigating through objects. The study predicted the sites success with audience groups and suggested wording changes to make the interface more descriptive of the actual contents. Once visitors used the Web site, they were likely to want many more objects to be available on-line and to make further enquiries at the Museum site.
Keywords: evaluation, ethnography, user research, audiences, interviews, interface design, usability testing
This paper presents a Web site evaluation with a difference. Its approach builds on audience and communications research of the past fifteen years and can be described in broad terms as semiotic, ethnographic and constructivist. User Insite has been applying these approaches to evaluation of Web sites since 1997, but the ideas on which they are based have a longer history. In communications research there has been theoretical discussion of the ways the contents of television and new media create and position audiences and audiences negotiate these positions to construct their own meanings. Examples are found in Reception research (Charlton & Bachmair, 1990), Cultural Studies and Audience Ethnography (Livingstone & Bovill, 1999; Howard, 1998; Ang, 1996; Buckingham, 1993; Moores, 1993; Lull, 1990) and Naturalistic studies (Lindlof, 1987).
My own research began with close descriptive studies of children viewing television in the 1980s, (Palmer, 1986). Audiences have always been interactive in their behavior and active in constructing meanings, but this was usually downplayed by the media industries in the production process . Now we have a new medium where the relationships between audience members and contents must be made visible in the production process and built in to the design. Research appropriate to the new Web medium seeks the patterns in users engagement with Web contents and relates this to the everyday contexts of users (sometimes referred to as audiences or visitors).
The Museums and the Web conference has been a leader in tracing the shape of developments in Web site evaluation. We are very pleased to be presenting our own work here, and continuing the discussion about evaluation research to improve museum Web sites.
Aims of the project
The aims of this research were:
HistoryWired: A few of our favorite things (http://historywired.si.edu) is described as an experimental program through which you can take a virtual tour of selected objects from the vast collections of the National Museum of American History. Here you'll have an opportunity to look at hundreds of museum artifacts, most of which are not currently on exhibit. The Web site uses a tiled interface, designed by Smartmoney.com, to give direct access to descriptions of over 400 objects. The text and associated materials were especially written for the interests of broad audiences who are not necessarily museum visitors.
Because History Wired is database driven, changes could be made to the site to reflect the actions and interests of audience members, as they became known through their actual uses and comments.
This paper describes the major patterns that emerged in a close study of thirty-one participants in eleven sessions. These patterns would be expected to recur in many engagements with the Web site, but there may be additional forms of engagement that have not yet been observed.
Between 12 and 19 March 2001, research sessions were conducted with a variety of visitor groups to the NMAH. The purpose of the sessions was to observe the spontaneous interests of users rather than setting predetermined tasks. This way, we could discover how the Web site shaped a variety of responses, and how users created their own experience from the many possibilities of navigation and content. An open-ended interview was included to ask questions of importance to the Web site creators, as well as to gain comment about matters arising in the observation from the users perspective.
Eleven interviews were conducted with naturally occurring visitor groups. This included a total of 31 people. Participants were selected from those in attendance at the museum at different times of day. Participants were approached and asked if they would be willing to look at a new Web site that was being developed for the museum. If they agreed, they were taken a short distance to a conference room, the Lemelson Center, from which they could still see out to the public areas. The test site was available on a laptop computer, and participants made their own decisions about who would use the mouse.
The two distinctive methods were as follows:
Sessions varied from 13 minutes to one hour, with larger groups taking a longer time. The average session time was 25 minutes. More than half came from other states, on a holiday or on a school trip. Two groups lived nearby and were on a break from work.
The following groups were involved in the research:
The observation sessions and interviews were taped and later transcribed. The transcriptions and observation notes together gave a full description of the session, and these accounts formed the basis of an interpretive analysis. A history of each session recorded the actual pages visited by each group as part of the Web site browser activity, but this data has not been included in the analysis.
Initial reactions and general comments
In the observation stages, everyone began by moving the mouse across the map. Only one person tested the site by himself; the others were with friends or family. Perhaps as a result of this, comments and talk accompanied the use of the Web site. The talk was sometimes quiet and directed and other times noisy and expressive. Many unsolicited comments were made about the interface during the observations. From the beginning all the comments were positive, even glowing. Examples were:
Int (sic) that cool?
Thats really neat
This is amazing it really is- you just open it all back up again
This is really something. Very impressive
In three of the sessions, people mentioned they could spend a lot of time or all day looking at the site.
In most sessions, one person would read out the titles of the boxes or images, or they would read them out together. When they saw an image, for example Inside the Lone Rangers Mask, or read a description, there was often laughter or personal comment. For example, in response to the knives displayed in a Yiddish shop sign: Wild! Dont think I can handle that.
Distinctive forms of engagement with the Web site
As with all communications media, History Wired appealed to particular kinds of audiences. The creators of the site wanted to present and describe objects in ways that would appeal to broad and non-specialist audiences. The innovative nature of the Web site interface also raised questions about how it would be put to use by different audience groups.
The following were the distinctive patterns of use that were observed.
Using the mouse to travel left to right across the map, returning to the left and reading across again.
This approach mirrored conventional reading and was only observed in one session. As her cursor moved, the user read pop-up labels but did not click and explore further images and information. When this possibility was pointed out towards the end of the session, she tried it a couple of times but mainly continued to read the map left to right . The woman was visiting the museum with older parents and was a grade school teacher, probably in her forties. She did not use the Internet at home, only at school in connection with teaching.
Quickly exploring the dimensions of the content, both the surface geography of the map and the information pages behind it
One group of six 8th grade students did this with great enthusiasm and speed, with statements of cool! accompanying their discoveries. The person with the mouse, a girl, began by exploring in circles within the main concept groups, in this case Home/clothing, then added selections of keywords at the top. They discovered there were categories within categories.
Once the back pages were discovered, which happened after a few minutes, they combined fast exploration of the map with clicking on back pages and further information pages whenever various members of the group indicated an interest. This group did not find the timeline or zoom features, but when they were told about them late in the session, they were so excited, they began another round of fast exploration.
Careful exploration of the Web site to answer particular interests.
Three of the visitor groups, who were with family members, including adult children, used the site in this way. One example of this was a father and son who looked at the site, then discussed an author, Stephen Ambrose, associated with military history. They spent most time on directed activity, selecting the keyword, Military as well as moving around the relevant sections. They quickly discovered and used the back pages in these thematic areas. As they explored the site (the adult son used the mouse), they commented to each other about what they were finding and what they wanted. They didnt find specific information about 101st Airborne, which was what they had come to the museum itself looking for that day, but they did find objects of interest to them.
A second family grouping who used the site in a similar way was a school-aged daughter and her mother. They were remarkable for the intense interest in reading every word of all the information about particular subjects such as the Star-Spangled Banner. They scrolled down to do this. Again, they were unhurried and talked about the subject matter as they went.
Experimenting with the navigation functions
A fourth approach was guided not so much by thematic content or personal interest as by selection of particular functions that people found interesting to use in themselves. One adult son with his mother explored the site mainly using the zoom function, the topic menu on the left hand side and the enlarge function. It took them some minutes at the beginning to find these and use them, but then they enjoyed the interface itself, choosing back pages for familiar content such as Kermit, Benny Goodman and the Gutenberg Bible.
Seeking the visual experience of objects
A fifth approach seemed to focus on inspection of the variety of visual images, yielding a close engagement with the objects. Two women friends in their twenties found the zoom and enlarge features early and systematically viewed objects of interest by reading out the titles in the zoom display, clicking for the information pages, reading the first paragraph (they didnt scroll down) and then enlarging the image. They talked about the objects themselves as they did this.
The different patterns of engagement show that History Wired provides a variety of possibilities for different audiences. Even those who have little experience with the Internet can explore the site systematically, gaining pleasure from the thumbnail images and the labels on the map. The intention of the designers, to provide direct experience of a large number of objects and to place in the hands of users many different ways of driving their own experience, has met with very positive responses. Some enjoyed the driving itself while others were more focused on content, or they could hop from image to image. This is a feature which is not characteristic of most Web sites. While the content may vary based on user selections, sites rarely provide different driving experiences.
Some of the functions, such as the zoom, timeline and drop-down menu, were not used by many. However, they were all used by some, to support their exploration of the site. Confident users, experienced with scrolling and drop-down menus, had much more depth of experience, and it was then their purposes or personal preferences for image or written text that seemed to determine the ways they used the site.
The site was very flexible in providing different reading experiences. There was an immediate engagement with the written descriptions, though most users read the first few paragraphs only. There were no difficulties with the reading, and three groups read all of the text for some of the objects they chose.
Responses to the interview question, What did you like best? confirmed the very positive comments made while people were exploring.
Its like walking around the museum, exactly the same way, its also better cos were not walking around
I enjoyed all of it
Its keeping up with technology we have today
Its really informative and original and itd be helpful I think
One boy in 8th grade who had not had a turn with the mouse made the observation:
I kind of like the way it was but I guess some people would want it simpler. My mother whos completely computer illiterate I think it would be hard for her.
Comparison with other Web sites
Interviewees liked the originality of the site and the associations it made.
Father: I've never seen a Web site like that. The format is different. Smaller or something.
Son: Its the first Web site [Ive] seen with so much stuff and a different way of finding it. Usually you just click. Theres a wide variety, saves a lot of time.
One person interviewed was an educator knowledgeable about Web site design:
If youre doing research on a particular thing, you can't search immediately. But its like you throw a dart at a dartboard, you see other avenues in trying to get a more well-rounded collection of information about this particular thing. Nice.
A college student observed:
Ive never seen another site that you can actually section up in little tiles like that before see it before you even open up the page, see what it relates to.
A favorite feature mentioned in four interviews was the connection between the map and the keywords along the top, shown by pointer lines connecting them. Another was the timeline, and the ability to access all objects at once.
Daughter: I like the pictures and the timeline.
Mother: I like the multiple references up at the top, especially as you cursor around, the reference points back up to bar. That's pretty nifty.
Im really impressed with the connectivity between choices, these lines coming down, works well for me, a kind of graphic orientation. The whole idea of highlighting boxes pertaining to category is very nice as well.
I like the way its set up where its all in one place. In a lot of places, like, you can still look at the site its simpler, like its all more compact.
People could name their favorite objects. Their answers revealed a connection between the favorites and their own interests. One grade 6 schoolgirl observed:
I liked the Log Cabin. Interesting. A bit of history. I like history a lot.
A nurse commented:
I liked the medicines. It's my job. Anything to do with medicine and medical history I like.
Pleasure in the objects was one of the major satisfactions of the Web site. This was observed rather than commented on in the interviews. As groups explored the front and back pages, they would read titles together, respond, comment and read further with interjections of cool. neat, Oh thats nice. Selection of the objects was successful in providing for the diverse interests of the group tested. The inclusion of some familiar objects enhanced their engagement with the site.
There were criticisms and suggestions for improvement of the site. The suggestion made by the majority of visitors was for the addition of a search facility. [The current version of the program now has a search facility.]
I think it would be a better research tool if you had a search engine.
Search. Make it easier What you are thinking might not be same as what the person who made this was thinking.
In three interviews, there was a request for additional information about objects: whether they were available to be seen; and a floor plan to show their exact locations.
Other suggestions included:
Groups were asked who the Web site was especially made for. Half of those interviewed saw it as useful for those who were planning a trip to the museum, or following up afterwards:
You could find a bunch of items that you want to see. When we came we didnt know what we wanted to see we had trouble deciding what we wanted to see.
If you were interested in something here you could see if you could find some more information from the Web site.
They defined the following types of audiences:
Its particular attraction for younger people was described:
Theres a cause and effect thing going on here. As soon as I move the cursor, a lot of things are happening which would be very engaging to school and high school age children.
History Wired was also thought to appeal to an audience who were interested in just exploring:
If ever youre not looking for something in particular it would be fun to mess around with it.
People were asked what they thought of the name and what it meant. This is not reported in detail because the original name will probably be used for future on-line exhibitions. It was discovered that the original name polarized the audience . Young people liked it, because it was dramatic and a bit daring. Older people rejected it because it could be misunderstood as an insult. It was uncanny that in one group the students thought up one of the potential names suggested by curatorial staff, and rejected it.
I think its cool because a lot of the names you see, it would be like American Artifacts. I think that would be really boring.
Participants could vote on how much they liked the description for each object that they viewed. A question about the voting option and whether participants would use it was asked in each interview because very few people used the voting function during the observations. Younger users seemed to like the voting option whereas those beyond school age were not interested, or said they would prefer to vote about something more meaningful like the quality of information.
I like it. You get to have an opinion on something. When you go to vote, it takes you to a screen about what percentage of people said yes and no. Will it do that?
Yes, I vote on Web sites. Like the pop culture and stuff. You vote if you think she's pretty or you think her dress is good. I feel like more of an individual
Im not likely to do it. At this level its sort of a teaser I would vote on the next level, the depth of information. Ive not invested much time personally at this point.
Didnt see it, I would vote if it referred to a description but not just if I like it.
In one family, the younger daughter said yes but the father said no and the adult son said he didnt care.
The interviews did not seek comments about every detailed feature of the Web site. However, where comments were made about specific features, they are listed below.
First I was lost, confused but after a while it was cool, once you figured out how it worked
The pop-up menus are nice. See what you are looking at. Neat that it overlaps, gets darker
Pretty easy to understand
I was confused about the black lines cos I know its all broken into categories is this all Communication?
Home and Clothing. (she demonstrates) This part is more like furnishings, such as bowls, and this ones more like accessories. You have Lincolns hat then you have a charm bracelet it would be better if it said what the divides mean.
Oh, these secondary screens have a nice clean look. Meaningful options. This is a pretty substantive lot of information. Here is where I'm more likely to go.
Once I started to use the further information, I got the idea.
That is so cool It has different varieties of all the information. Moves around.
I like it, its not so busy. Like taking the map and simplifying it a lot.
Headings? Good because you can narrow it down to what your interest is. It makes it a lot easier to use.
Audience and content development
The questions about History Wired concerned the ways in which the Web site attracted particular kinds of audiences, and the ways these audiences engaged with the site. Over time, new audience relationships will be created by the sites content and innovative design.
The detailed study of audience/content relationships suggests further strategies to develop content and extend the audiences for the Web site. Because these strategies are based on current interactions of audience members with the site, they are likely to succeed.
Professionals in the image-related industries (media, publishing, architecture, graphic arts, etc)
NMAH currently receives many requests to librarians and curators about images of objects it possesses, to be used in publications and other communication. History Wired is likely to attract the interest of such professionals.
Genealogists and family historians
People interested in family history are a very large group with strong international research links and excellent Internet skills. They are usually over 45 years of age, with time to research and explore. Contents on History Wired about particular families, or about transportation connected with major immigration movements, are likely to attract this audience.
International audiences and those with limited skills in reading English
The site will reward those who cannot read English language text well, but who learn to navigate and find larger examples of the thumbnail images. The text, written for a general audience, will also be more engaging and will reward the effort of reading. This site is likely to attract an international audience, especially those interested in museums, historic objects, or American history who have previously not accessed NMAHs on-line exhibitions because of language difficulties. The site will also be welcomed by users with an intellectual disability because the content is complex and interesting to adults but can be accessed with the movement of a mouse. Larger images can be found by responding to thumbprint images, without the need to read captions.
Some children in grade school will enjoy features of this site, especially the way they can drive it, fast, with something happening visually in response. This evaluation included only one early grade-school girl who was intensely interested in the Star Spangled Banner and seemed to read every word. It would be worthwhile to consider a special study of children as audiences for this site, so the sites educational uses can be enhanced.
Collectors, clubs and special interest groups
A number of different groups will be drawn by specialist content which is closely related to their leisure or professional occupation and which opens up a resource of information and images not currently available. For example, auctioneers and retailers of antiques and historical objects may use History Wired to make comparisons with their own objects for sale.
Special collections and exhibitions
The drop-down menu function gives the possibility of creating special collections as part of History Wired, or featuring new content such as mystery objects. These items would attract existing collectors and new audiences. A collectors topic could be defined in the drop-down menu, then key words could be used to highlight items as collectors define them; for example coins, stamps, football cards.
These parts of the site invite the development of discussion groups or on-line donation and sharing of images and information. This might also be a location for creating special exhibitions or temporary collections.
Dialogue with curators and experts
With the current objects, it may be exciting to make available on-line the kind of expert advice from curators and collectors that visitors often seek in the physical museum. For limited periods of time, curators could be available to respond to e-mail questions and comments on particular museum objects. This would be highlighted on the information pages for each object. The interchange between curators and the public would also assist the NMAH to understand how users were engaging with the Museum in new ways through this on-line medium.
One of the distinctive qualities of History Wired is the provision of different means to access, view and engage with the large collection of objects. At the same time, there is a consistency of address to the reader; one which tries to engage interest in the special stories or features associated with the object. An analysis of the content would have delineated the features of the writing, language , images and their interrelationship that created the possibilities for engagement that were taken up so enthusiastically by audience members.
Making the different audiences visible in this research has confirmed the importance of the careful work by producers and writers, as well as the significance of the experience that on-line visitors can create from material designed for them.
Research based on ethnographic approaches promises to contribute more to Web site evaluation than a diagnosis of the functioning of the site and a judgment about whether it is user friendly. Attention is paid to the emerging relationships that audiences create in their engagement with the site and its navigational and content possibilities. Relating this activity to the everyday contexts of audiences then suggests new developments such as extending the contents to satisfy specialist interests or to effectively address audiences whose uses were not originally predicted. The focus on audiences constructing meanings leads to interpretation work on-line that creates more extensive use and deeper understanding by Web site users.
This research was supported by an Australian Fulbright Professional Award. It was conducted within the Web Program at the NMAH with the excellent guidance and support of Dr. Judy Gradwohl, Manager, as well as Matthew MacArthur and Curtis Prather.
Ang, I. (1996). Living Room Wars. London: Routledge.
Buckingham, D. (Ed.) (1993). Reading Audiences. Manchester: Manchester University Press.
Charlton, M. and Bachmair, B, (Eds.) (1990). Media Communication in Everyday Life. Interpretative Studies on Childrens and Young Peoples Media Actions. Munich: K.G . Saur.
Howard, S. (Ed.) (1998). Wired-up. Young People and Electronic Media. London: UCL Press.
Lindlof, T. (Ed.) (1987). Natural Audiences: Qualitative Research of Media Uses and Effects. New Jersey: Ablex.
Livingstone, S & Bovill , M. (1999). Young People New Media. London School of Economics and Political Science.
Lull, J. (1990). Inside Family Viewing. Ethnographic Research on Televisions Audiences. London: Routledge.
Moores, S. (1993). Interpreting Audiences. London: Sage.
Palmer, P. (now Gillard) (1986). The Lively Audience. Sydney: Allen & Unwin.