Archives & Museum Informatics
158 Lee Avenue
Toronto, Ontario
M4E 2P3 Canada
info@ archimuse.com
www.archimuse.com

Published: March 15, 2001.

"Less Clicking, More Watching": Results from the User-Centered Design of a Multi-Institutional Web Site for Art and Culture

John Vergo, Clare-Marie Karat, John Karat, Claudio Pinhanez, Renee Arora, Thomas Cofino, Doug Riecken, Mark Podlaseck IBM T.J. Watson Research Center, USA

Abstract

This paper summarizes a 10-month long research project conducted at the IBM T.J. Watson Research Center aimed at developing the design concept of a multi-institutional art and culture web site. The work followed a user-centered design (UCD) approach, where interaction with prototypes and feedback from potential users of the web site were sought throughout all the phases of the design process. In the first phase of the research, we conducted interviews with museum curators, issued questionnaires to brick and mortar museum visitors, performed on-line surveys with users of museum web sites, ran focus groups, and performed usability walk-throughs of best-of-breed museum web sites and of our new web site design ideas.

The results showed a surprisingly strong interest in streaming multimedia, TV-like experiences where users watched experts and artists talk about art and culture, augmented by links to additional and in-depth information. Such a design strategy was clearly favored by our subjects over more interactive experiences such as chat rooms, information search systems, or user-curated tours. In other words, when users go to a web site of arts and culture looking for entertaining and educational content, they seem to be less interested in clicking to find information than in watching people, especially experts and celebrities, present and discuss artistic works from their perspective.

Based on the results of the first phase, we developed a prototype of a web site based on the concept of "tours" --- 5-10 minute streaming multimedia experiences led by experts and artists, accessible by users with normal telephone connections (56Kbps modems). The tours resemble a slide show enhanced by narration, music, occasional video clips, and hot spots for extra information. Two pilot tours were fully produced and tested with users who reported great satisfaction with the tours in terms of their entertainment, engagement, and educational values. We also observed that the most satisfied users were those who interacted and explored the least during a tour.

The research described in this paper suggests that people interested in art and culture have an affinity for web sites where they can have entertaining, curated experiences in many ways similar to short TV-like documentaries. Yet there are many other occasions when they are looking for engaging educational and entertaining experiences. In such situations, less interactive but narratively structured streaming experiences with a human voice seem to be a very appealing format to engage and satisfy such users. Importantly, they do not see such experiences as a substitute for actual visits to museums or attendance at live performances, but as an enriching and highly accessible way to augment the cultural experiences and performances they enjoy in brick and mortar cultural institutions around the world. People are often looking for specific information about artists and art works.

Introduction

In the summer of 1999, IBM decided to investigate the possibility of sponsoring and creating a multi-institutional web site for art and culture. The goal of this web site is to give users around the world ways to increase their appreciation of artistic works, cultural treasures and performances by providing information and learning opportunities about art and culture.

User centered design

User-centered design (UCD) is a design approach in which multidisciplinary development teams create useful, usable and satisfying user experiences. Central to UCD is the notion that the design process is iterative, and is accomplished by nearly continual input from users. As user input is collected and analyzed, it is fed back into the design process, resulting in progressive refinement and improvement of the system under design.

We followed six general UCD (IBM, 2000) principles which guided our activities:

Set goals for the project. This early phase of the project is also the right point to spell out any competitive goals, along with an identification of the primary competitors.
Understand users. Define the intended user population and the context of use. The context of use includes the physical context and the social context, if both are relevant.
Assess the competitiveness of the system under design. If the goal is to design a system that breaks new ground, it is risky to attempt such a design without a thorough understanding of the competitive landscape. Having clearly stated goals for the intended user allows testing of the system against competitive alternatives and comparing the results.
Design every aspect of the user experience. Everything a user sees and touches is designed together by a multidisciplinary team. In the case of the design of a web site for art and culture, the user experience goes well beyond the content to include navigating the site, interacting with virtual exhibits, and even downloading and installing browser plug-ins.
Create and evaluate designs iteratively with representative samples of the targeted user population. User feedback is gathered early and often, and this feedback drives design and development. In the early phases of design, lower fidelity prototypes are presented to and tested with the users. As the design team's understanding of the effectiveness of various design alternatives and design elements is clarified, higher fidelity prototypes are developed and tested. This allows inexpensive testing of many ideas early in the design process. Once a design idea (or set of design ideas) tests well with a low fidelity prototype, it is less risky to commit development dollars and resources in pursuit of the idea(s).
Manage the project through continual user observation and testing. Throughout the life of the project, continue to monitor and listen to your target user group, and let the feedback inform any modifications to the system design.

These guidelines are a set of heuristics that we typically follow on any project. They provide tremendous flexibility with respect to the precise activities a project team might employ and include a wide range of established user research methods and participatory design methods in use today. The next section explains the specific techniques we employed on our project, along with the rationale for our choices. After each UCD activity is presented, we report some of the most significant findings resulting from the activity. Each activity yielded a wealth of data not presented in this paper but publicly available (Karat 2001, Pinhanez 2001). Those findings that support our ultimate design choices are presented, along with findings of general interest to the Museums and the Web community.

Our UCD Approach

Project goals

The initial requirements for the project were extremely broad and open. We set out to create on the Web a major cultural destination (not a portal) that would attract a large number of visitors. The site had to provide access to content supplied by 5-10 of the largest and best-known cultural institutions around the world. The institutions were chosen to represent different areas of art and culture, including various performing arts, fine arts, natural history and science. Finally, each exhibit on the site needed to combine content from multiple institutions, to leverage the powerful institutional partnerships and to ensure the uniqueness of the site.

An additional goal was that the web site must not be a database of cultural artefacts or knowledge, but instead it must attract users by enabling entertaining and educational experiences similar to those provided by visiting a museum, attending a performance, or watching a cultural TV program. The success of the web site would be measured by its popularity and, specifically, by its rate of return visits. Given the openness of the initial specifications of the project and the strong requirement of users' satisfaction, we adopted a user-centered-design (UCD) approach to the development of this web site.

We ran five major user research activities in the discovery phase of the project. We conducted interviews with museum curators, issued questionnaires to brick and mortar museum visitors, performed on-line surveys with users of museum web sites, ran focus groups, and performed usability walk-throughs of best-of-breed museum web sites and of our new web site design ideas. Before engaging in any of the user research activities, we took the time to define our target user population.

Defining the target user population

Our definition of the target users for our project was initially shaped by the project business goals. The sponsoring organization within IBM came to us with some initial parameters. We eventually defined our target users to include members of several different population segments:

Adults 21-34 years old,
Adults 35-50 years old, with and without children
Adults 50 - 70 years old, and
Children 9-15 years old, typically accompanied by adults from segment "2", above

While we did not target people 16-20 years old, we did not explicitly exclude them from our user population.

We assumed that our typical users spend an average of 10 or more hours a week on a computer, and of that time, five or more hours are spent on the Internet. They find out about sites from search engines, advertisements, friends and family, and web surfing. They think that having cultural events as a part of their lives is somewhat to very important. We also anticipated that visitors see the site as a way of enriching their experience at the brick and mortar center, not as a substitute for it.

Focus groups

Our first project activity was to run a series of six focus group sessions, which were conducted on the east coast (New Jersey), mid-west (Michigan) and west coast (California). Each group consisted of eight individuals, gender balanced. Two groups of age groups 21-35, 35-50 and 50-70 were conducted. An independent, objective third party moderated the sessions.

Some of the initial ideas that were explored during the focus group session were:

Giving users the ability to create their own exhibit
Live virtual tours where artists or curators are available on-line, in real time, to conduct a tour
A cultural community center allowing users to meet, talk and exchange ideas on a wide rage of cultural topics
A site that could be personalized to an individual's cultural interests, so that an individual visiting the site any time sees cultural content that is more likely to have personal interest

During the focus group sessions, participants were presented with verbal descriptions and simple visual representations of the new design ideas, and were asked a wide range of questions with the following objectives:

Gaining a better understanding of the target audiences' use of cultural resources.
Uncovering consumers' experiences with existing cultural web sites.
Understanding current web site usage and behaviour.
Assessing reactions to the proposed web site features/activities.
"Brainstorm" potential features for inclusion on the web site.
Determining relevance of and interest in the web site concept.
Understanding the receptivity to various payment methods (i.e. advertising, sponsorship, subscriptions/memberships, etc).

Focus group results

Overall, the newly proposed web site features sparked favorable reactions from consumers. "Live" tours were overwhelmingly considered one of the most compelling and unique ideas proposed. They particularly liked the opportunity to interact with artists and tour guides.

Respondents were adamant that they should not be charged a fee to use the features offered on the web site. Advertising banners and third-party sponsorships were deemed the most acceptable means of funding the site. However, it must be noted that the use of advertising banners was a forced choice response. In the usability walkthroughs, people were clear in their disdain for advertising banners.

Most respondents viewed the newly proposed web site as a supplemental source of information and/or enhancement to the actual live cultural activity or event. While east-coast participants usually use ticketing and venue web sites to gather logistical information regarding a cultural activity, mid-west and west-coast respondents also venture to cultural "home sites" for background information relating to the activity or event. Most respondents conduct some form of preparatory research before attending a cultural activity or event.

Time, geographic limitations and financial constraints are the primary factors that hinder respondents from attending cultural events as often as they would like.

Curator interviews

We conducted a series of interviews with curators from two partner institutions. Five curators were interviewed, with each session lasting between one and two hours.

The interviews were conducted with a specific set of goals but did not follow a rigid script. We queried curators to understand the design criteria for exhibits. We were interested in understanding if there are differences between institutions on how design criteria are created and implemented. We asked them how the design criteria affect the final exhibit. We wanted to know if curators go through the explicit step of defining the target audience for each exhibit. Finally, we asked the curators how they (or their management) measure the success of their work.

Curator interview results

Curators at both partner institutions stated that most individual exhibits are attractive to narrow audiences, and that many different types of target audiences exist. Curators from both institutions expressed an interest in drawing new people (i.e. new visitor segments) to the institution. Curators at an institution indicated they specifically target 20-30 year old visitors on a regular basis with the goal of establishing a satisfied clientele that will engage in repeat visits to the institution for many decades to come. Both institutions consider "families with children" a target audience.

All institutions have "dead zones", times when visitation is slow, and they try to draw people in to the museum by putting on exhibits that have appeal to a wide audience. The curators reported that their institutions perceive a strong need to keep exhibits "fresh", in order to attract repeat visitors.

No institution employs a quantitative approach in measuring success other than simply counting visitors. The institutions use ad hoc observational techniques in the museums. They also gauge reactions through published reviews in magazines and periodicals.

Usability walkthroughs

The usability walk-throughs were run in 12 sessions with a total of 70 participants who were screened for cultural interest and basic familiarity with the web. Subjects were first shown excerpts of existing web sites related to culture. Groups of approximately six subjects each were assembled in the TJ Watson Usability Lab in Hawthorne, NY. As much as possible, the groups were composed of 50% women, 50% men, and represented the diversity in income, education, background, race, and ethnicity in the tri-state area.

Each group was shown sections of two existing cultural web sites (children were shown only one site), followed by two new design ideas for the proposed site. After each demo, a facilitator led a group discussion of their opinions of the content in the demo. Then participants recorded their individual thoughts about the demo content (children had no written part). Then the next demo was shown. At the end of the adult session, participants were asked to answer several summary questions (children did not do this part). Over the course of all the walkthroughs, five existing web sites were shown and five new design ideas were presented.

In the first part of the session, we showed existing web sites where users could navigate through 3D replicas of exhibitions, interact with panoramic images, visually explore paintings and 3D sculptures, write and publish stories about the content, and obtain information about different layers of a painting.

The mockups of our new design ideas shown in the second part of the usability walk-throughs included five different possibilities for exploring cultural content:

A filtering system based on direct manipulation of large databases with visual feedback (such as in Alberg & Shneiderman, 1994);
A set of lenses (tools) to manipulate the way content could be viewed (such as in Stone, Fishkin, and Bier, 1994);
A chat system where people could talk about a particular artwork (Viegas and Donath, 99)
A notebook system where the user collects and comments on artistic content, and later publishes the notebook for public/private viewing;
A multimedia system where the user watches guided multimedia tours, interacting whenever interested in related information.

Usability walkthrough results

A major finding of the usability walk-throughs was that most of the participants did not express interest in web sites that involved active interaction with the content or other people, such as when using a filtering system, creating a notebook, or chatting. The multimedia prototype was clearly the best received among the design ideas. Among the existing web sites, there was a preference for sites where the user was guided through an experience or discovery process, and in this case, participants strongly suggested the replacement of text by audio.

We summarize those findings by hypothesizing that in this kind of entertaining web experience, users want less clicking, more watching. Users seem to be very comfortable with the idea of a streaming web experience that leads them through artistic and cultural artefacts where, unlike television, the stream can be paused, replayed, or interrupted for further exploration. In fact, we found a strong desire for availability of additional information through hypermedia links and in-depth analysis of the works of art. Interestingly, some of the participants viewed the more interactive design concepts and existing web sites as work, not entertainment.

The idea of more passive experiences was further reinforced by many complaints about the difficulty of 3D interfaces, the excess of textual information, and the amount of work required to find information. The streaming multimedia concept seems to address this major concern by neither requiring the user to learn and exercise a large set of new skills nor requiring a large amount of attention and decision-making.

The usability walk-through also detected a desire for a "human voice" behind the multimedia experience; that is, a personal viewpoint in the exposition of the content. However, this voice needs to have the authority of an expert in the field or the human interest of a celebrity. In fact, a major complaint identified with chat systems was precisely the lack of trust and perceived value in the information provided by generic chatters.

We explicitly asked the usability walkthrough participants what factors would be important in their decision to re-visit the site on multiple occasions. The most frequent responses to this question were:

The quality of the content
The ease of use of the site
Having a cultural calendar that would send them reminders;
The quality of the images, video and audio
Having new exhibits
Access to archived old exhibits.

The modal response to the question "How often should the site be updated with new material" was one month, but there was considerable variance in the data.

Web site surveys

The web surveys were conducted by using a questionnaire that automatically popped up, in a separate window, whenever users accessed the home page of two partner art museums. Answering the 17 questions was optional, but throughout the month the survey was conducted, 830 and 1417 users answered the survey on each of the museum web sites. The survey was centered on questions about how the respondents use the web to obtain art and culture-related information and entertainment.

Web site survey results

About 81% of the surveyed visitors were visiting the web sites for the first time. About half of them (45%) got to the site through search engines and 17% through links from related sites. An estimated 10% came directly from other web pages.

About 25% were researching information on specific content, while 32% were interested in specific information about the museum (schedules, access, etc.). About 12% were looking for fun-like activities and only 2.4% declared interest specifically in shopping.

The absolute majority of the visitors (90%) were alone while visiting the sites. The most important aspects in the visit were the "content" (46%) and the ability to pre-visit the museum (31%). Only 14% ranked "usability" as their major concern.

About half of the visitors considered the sites "interesting", 21% saw them as "educational", and only 4% thought of the sites as "entertaining". This, in some ways, correlates with the objectives of the visitors as described above.

Visitors considered actually getting the information they wanted as the most engaging part of the experience of the site (39%). Similarly, when asked what part of the experience could not be replicated, 50% of the visitors mentioned the facility of getting information, and 35% mentioned the convenience of visiting the museum from home. The biggest frustrations with the web sites were

The inaccessibility of the full content of the museum (35%)
Not finding the sought-for information (16%)
The inability to access the schedule (9%)
The inability to ask questions of the curators (5%)

There was a slight predominance of males (58%) among visitors to the site. The age distribution was quite homogeneous, with a strong participation of people in their forties (22%) and fifties (11%).

Most of the visitors (58%) belonged to families having more than 5 hours/week of web usage and only 22% used the web less than 2 hours/week. As for connection time, 76% had a connection equal to, or faster than, 56Kbps, and 99% have a 28.8Kbps or better connection available.

To the users who answered the survey, the most popular education web sites were Discovery (6%), Yahoo, PBS, and the New York Times. The most popular entertainment web sites were Yahoo (6%), the Internet movie database, ESPN, and the onion.

Brick and mortar visitor surveys

A total of 99 visitor surveys were conducted at two of our partner institutions. The surveys at each institution were conducted on multiple days in order to get information from weekend and weekday visitors, as well as to survey individuals who might be visiting to attend a special event at the institution.

Brick and mortar visitor survey results

People enjoy integrating cultural experiences into their activities in order to create a rewarding social, entertainment, and educational experience - also, visitors to the institution are interested in learning and enriching themselves and their friends and families.

Many people selected the institution they were visiting because it provided an event that would satisfy both social and educational goals. A visit could also be integrated with other social events (visitors appeared to schedule a trip to the institution along with other activities such as a walk in the park, shopping, and dining). A majority of visitors visited to enrich themselves by viewing exhibits and special events. Many visitors indicated that the use of different presentation techniques (e.g., kiosks, films, human guides, lighting effects, etc.) greatly enhanced the learning experience.

The nature of the content dictates whether people are more focused on entertainment than education, and is primarily responsible for shaping the experience. As for the design of the web site, these surveys point out that there should be different treatments and different goals for different kinds of content. For instance, scientific content asks for an experience designed to be more educational than entertaining, while content related to comedy and drama should be entertaining and emotionally engaging.

There is a significant difference between the normal visitors and the audience for special events. Special events are less "social" than regular visits. In terms of the design of the site, this data should inform the number of community-related features (for instance, chatting) available for regular browsing visitors versus the audience for special events.

Conclusions of the Discovery Phase UCD activities

From the responses of the web surveys, we found that only a minority of people (4 to 5%) label the experience they have at museum web sites as "entertaining". In many ways, the participants in our research seem to lean towards defining an entertaining web experience as something closer to traditional TV, but enriched by the opportunity to explore and find extra information. However, our research also indicated that to access the majority of our targeted audience today, such web experiences must be available to users sporting modems of 56Kbits/second or more.

Based on the results of the discovery phase, we developed a design concept for the cultural web site based on the idea of multimedia tours guided by experts and artists. In our design, a tour presents information to the user continuously, from beginning to end, unless the user signals a desire to explore extra material or to exercise control. During the tour, opportunities to obtain additional information are presented as hot spots on the screen. All of the extra information is also available for exploration at the end of the tour.

The proposed web site consists of a collection of tours covering different areas of art and culture, such as painting, music, theater, and sculpture. In order to satisfy a large number of users, it seems to be necessary to provide a large number of tours and to add new ones regularly. Some of the design decisions presented below are, therefore, the result of the need to reduce production costs.

To cope with the requirement of maximum 56K bandwidth, we decided to explore multimedia experiences primarily based on still pictures and sound, instead of low quality video. At 56Kbits/second, video tends to become small and blurry, while at that speed it is possible to download audio with reasonable quality and reasonably good-looking pictures. Dealing mainly with stills also serves to reduce production costs, since shooting video is normally quite a bit more expensive than just audio recording and picture taking.

In our design, the main multimedia experience, or simply main tour, is enriched by the addition of user controls such as pause/resume/rewind/forward and by the inclusion of hot spots for two kinds of extra content: side tours and branches. A side tour is an extra, self-contained segment of multimedia, normally focusing on more specific aspects of the tour subject, which can be "inserted" into the main tour. A branch is a static web page with text, pictures, and links to extra information on a specific subject. The decision to create a side tour instead of a branch is based on the expected reach of the content. Since side tours are much more costly to produce than branches, we tend to produce side tours only for highly desirable extra information.

Figure 1 shows a snapshot of a tour. Most of the area is used by the content being presented by the tour (pictures, text, occasionally very short segments of video). On the bottom left side, a pictorial map gives the user a basic idea of the duration of the different scenes of the tour and the elapsed time. Rolling the mouse over the map brings extra textual information about each scene, while clicking on the picture of a scene interrupts the current scene and immediately starts the scene corresponding to the clicked image.

Figure 1 Typical scene of a tour with its navigation map, including links to a side tour and two branches

As the tour progresses, hot spots indicating side tours and branches appear on the screen. When a side tour is selected, the main tour is interrupted and the side tour is played. When the side tour finishes, the main tour automatically resumes from the point where it was left. A click on a branch pauses the tour and opens a new window on the browser, displaying the web page associated with the branch. To resume the playing of the main tour, the user is required to click on the pause/resume marker above the map.

When the tour finishes, all the content - main tour, side tours, branches - of the tour becomes easily available on the explore page. Figure 2 depicts the explore page associated with the tour offig. 1. Clicking on the tour map restarts the tour from the beginning of the scene that was clicked. Similarly, clicking on side tours and branches immediately starts them. The explore page is automatically presented to the user at the end of the main tour. However, during any moment of the main tour, the user can access the explore page by clicking on the corresponding hot spot on the right of the map (fig. 1).

Figure 2 Exploration page that allows access to the main tour and all side tours and branches.

The Two Pilot Tours

The design concept described above instructed the development of two pilot tours that are the main instruments used in the evaluation methods later described by Pinhanez and Karat (Pinhanez 2001, Karat C. 2001, Pinhanez 2001).

The first tour features the work of a toy pianist, Margaret Leng Tan. In the tour, the pianist talks about her involvement with toy pianos, how music is arranged for a toy piano, and her connections to Schröeder, the famous cartoon character created by Charles Schulz. Two side tours describe the history and functioning of toy pianos and the work of Margaret Leng Tan before becoming a toy pianist. The main tour lasts 4m15s, and the side tours take 1m18s and 50s, respectively. The tour also includes five branches.

The second tour talks about Ludwig van Beethoven and his Ninth Symphony. In this tour, a narrator leads the user through aspects of Beethoven's life and work, with comments also by a maestro and a concert pianist. Three side tours are provided, one about Beethoven's deafness and two others that let the user explore Beethoven's scores and his famous Heiligenstaden testament. Beethoven's main tour lasts 10m10s and the first side tour 2m00s. The other side tours, since they incorporate interactive elements, have no fixed duration, although their exploration usually takes around 60s each. There are also 5 branches available for user exploration.

After completing the development of the prototypes, we again engaged a sample of our target user population in a usability study to assess the success and value of our designs. The prototypes were tested with men and women from 21-55 years old. Test results indicate high user satisfaction with the tours. Users interacted relatively infrequently with the tours, and the less they interacted, the more they reported feeling engaged and entertained by the experience. This supports our initial hypothesis that the tours are most entertaining when they are experienced as they were designed to be; that is, watched as a TV-like experience. The results of the usability study give support to the contrarian "less clicking, more watching" design approach identified in the discovery phase.

References

Alberg, C. & Shneiderman, B. (1994). Visual information seeking: Tight coupling of dynamic query filters with starfield displays. Proceedings of CHI '94 (Boston MA, April 1994), ACM Press, 313-317.

IBM, 2000, http://www-3.ibm.com/ibm/easy/eou_ext.nsf/Publish/13

Karat, C. et al. (2001), Less Clicking, More Watching: Results of the Iterative Design and Evaluation of Entertaining Web Experiences. Submitted to Interact'01.

Pinhanez, C. et al. (2001), "Can Web Entertainment Be Passive", Submitted to International World Wide Web Conference, Hong Kong,

Pinhanez, C. et al. (2001), "Less Clicking, More Watching": An Option for Entertainment on the Web?" Submitted to CHI'01, Seattle.

Stone, M., Fishkin, K., & Bier, E. (1994). The movable filter as a user interface tool. . Proceedings of CHI '94 (Boston MA, April 1994), ACM Press, 306-312.

Viegas and Donath (1999), Chat Circles Proceedings of CHI '99 ACM Press.