Published: March 15, 2001.
"Less Clicking, More Watching": Results from the User-Centered Design of a Multi-Institutional Web Site for Art and Culture
John Vergo, Clare-Marie Karat, John Karat, Claudio Pinhanez, Renee Arora, Thomas Cofino, Doug Riecken, Mark Podlaseck IBM T.J. Watson Research Center, USA
This paper summarizes a 10-month long research project conducted at the IBM T.J. Watson Research Center aimed at developing the design concept of a multi-institutional art and culture web site. The work followed a user-centered design (UCD) approach, where interaction with prototypes and feedback from potential users of the web site were sought throughout all the phases of the design process. In the first phase of the research, we conducted interviews with museum curators, issued questionnaires to brick and mortar museum visitors, performed on-line surveys with users of museum web sites, ran focus groups, and performed usability walk-throughs of best-of-breed museum web sites and of our new web site design ideas.
The results showed a surprisingly strong interest in streaming multimedia, TV-like experiences where users watched experts and artists talk about art and culture, augmented by links to additional and in-depth information. Such a design strategy was clearly favored by our subjects over more interactive experiences such as chat rooms, information search systems, or user-curated tours. In other words, when users go to a web site of arts and culture looking for entertaining and educational content, they seem to be less interested in clicking to find information than in watching people, especially experts and celebrities, present and discuss artistic works from their perspective.
Based on the results of the first phase, we developed a prototype of a web site based on the concept of "tours" --- 5-10 minute streaming multimedia experiences led by experts and artists, accessible by users with normal telephone connections (56Kbps modems). The tours resemble a slide show enhanced by narration, music, occasional video clips, and hot spots for extra information. Two pilot tours were fully produced and tested with users who reported great satisfaction with the tours in terms of their entertainment, engagement, and educational values. We also observed that the most satisfied users were those who interacted and explored the least during a tour.
The research described in this paper suggests that people interested in art and culture have an affinity for web sites where they can have entertaining, curated experiences in many ways similar to short TV-like documentaries. Yet there are many other occasions when they are looking for engaging educational and entertaining experiences. In such situations, less interactive but narratively structured streaming experiences with a human voice seem to be a very appealing format to engage and satisfy such users. Importantly, they do not see such experiences as a substitute for actual visits to museums or attendance at live performances, but as an enriching and highly accessible way to augment the cultural experiences and performances they enjoy in brick and mortar cultural institutions around the world. People are often looking for specific information about artists and art works.
In the summer of 1999, IBM decided to investigate the possibility of sponsoring and creating a multi-institutional web site for art and culture. The goal of this web site is to give users around the world ways to increase their appreciation of artistic works, cultural treasures and performances by providing information and learning opportunities about art and culture.
User centered design
User-centered design (UCD) is a design approach in which multidisciplinary development teams create useful, usable and satisfying user experiences. Central to UCD is the notion that the design process is iterative, and is accomplished by nearly continual input from users. As user input is collected and analyzed, it is fed back into the design process, resulting in progressive refinement and improvement of the system under design.
We followed six general UCD (IBM, 2000) principles which guided our activities:
These guidelines are a set of heuristics that we typically follow on any project. They provide tremendous flexibility with respect to the precise activities a project team might employ and include a wide range of established user research methods and participatory design methods in use today. The next section explains the specific techniques we employed on our project, along with the rationale for our choices. After each UCD activity is presented, we report some of the most significant findings resulting from the activity. Each activity yielded a wealth of data not presented in this paper but publicly available (Karat 2001, Pinhanez 2001). Those findings that support our ultimate design choices are presented, along with findings of general interest to the Museums and the Web community.
Our UCD Approach
The initial requirements for the project were extremely broad and open. We set out to create on the Web a major cultural destination (not a portal) that would attract a large number of visitors. The site had to provide access to content supplied by 5-10 of the largest and best-known cultural institutions around the world. The institutions were chosen to represent different areas of art and culture, including various performing arts, fine arts, natural history and science. Finally, each exhibit on the site needed to combine content from multiple institutions, to leverage the powerful institutional partnerships and to ensure the uniqueness of the site.
An additional goal was that the web site must not be a database of cultural artefacts or knowledge, but instead it must attract users by enabling entertaining and educational experiences similar to those provided by visiting a museum, attending a performance, or watching a cultural TV program. The success of the web site would be measured by its popularity and, specifically, by its rate of return visits. Given the openness of the initial specifications of the project and the strong requirement of users' satisfaction, we adopted a user-centered-design (UCD) approach to the development of this web site.
We ran five major user research activities in the discovery phase of the project. We conducted interviews with museum curators, issued questionnaires to brick and mortar museum visitors, performed on-line surveys with users of museum web sites, ran focus groups, and performed usability walk-throughs of best-of-breed museum web sites and of our new web site design ideas. Before engaging in any of the user research activities, we took the time to define our target user population.
Defining the target user population
Our definition of the target users for our project was initially shaped by the project business goals. The sponsoring organization within IBM came to us with some initial parameters. We eventually defined our target users to include members of several different population segments:
While we did not target people 16-20 years old, we did not explicitly exclude them from our user population.
We assumed that our typical users spend an average of 10 or more hours a week on a computer, and of that time, five or more hours are spent on the Internet. They find out about sites from search engines, advertisements, friends and family, and web surfing. They think that having cultural events as a part of their lives is somewhat to very important. We also anticipated that visitors see the site as a way of enriching their experience at the brick and mortar center, not as a substitute for it.
Our first project activity was to run a series of six focus group sessions, which were conducted on the east coast (New Jersey), mid-west (Michigan) and west coast (California). Each group consisted of eight individuals, gender balanced. Two groups of age groups 21-35, 35-50 and 50-70 were conducted. An independent, objective third party moderated the sessions.
Some of the initial ideas that were explored during the focus group session were:
During the focus group sessions, participants were presented with verbal descriptions and simple visual representations of the new design ideas, and were asked a wide range of questions with the following objectives:
Focus group results
Overall, the newly proposed web site features sparked favorable reactions from consumers. "Live" tours were overwhelmingly considered one of the most compelling and unique ideas proposed. They particularly liked the opportunity to interact with artists and tour guides.
Respondents were adamant that they should not be charged a fee to use the features offered on the web site. Advertising banners and third-party sponsorships were deemed the most acceptable means of funding the site. However, it must be noted that the use of advertising banners was a forced choice response. In the usability walkthroughs, people were clear in their disdain for advertising banners.
Most respondents viewed the newly proposed web site as a supplemental source of information and/or enhancement to the actual live cultural activity or event. While east-coast participants usually use ticketing and venue web sites to gather logistical information regarding a cultural activity, mid-west and west-coast respondents also venture to cultural "home sites" for background information relating to the activity or event. Most respondents conduct some form of preparatory research before attending a cultural activity or event.
Time, geographic limitations and financial constraints are the primary factors that hinder respondents from attending cultural events as often as they would like.
We conducted a series of interviews with curators from two partner institutions. Five curators were interviewed, with each session lasting between one and two hours.
The interviews were conducted with a specific set of goals but did not follow a rigid script. We queried curators to understand the design criteria for exhibits. We were interested in understanding if there are differences between institutions on how design criteria are created and implemented. We asked them how the design criteria affect the final exhibit. We wanted to know if curators go through the explicit step of defining the target audience for each exhibit. Finally, we asked the curators how they (or their management) measure the success of their work.
Curator interview results
Curators at both partner institutions stated that most individual exhibits are attractive to narrow audiences, and that many different types of target audiences exist. Curators from both institutions expressed an interest in drawing new people (i.e. new visitor segments) to the institution. Curators at an institution indicated they specifically target 20-30 year old visitors on a regular basis with the goal of establishing a satisfied clientele that will engage in repeat visits to the institution for many decades to come. Both institutions consider "families with children" a target audience.
All institutions have "dead zones", times when visitation is slow, and they try to draw people in to the museum by putting on exhibits that have appeal to a wide audience. The curators reported that their institutions perceive a strong need to keep exhibits "fresh", in order to attract repeat visitors.
No institution employs a quantitative approach in measuring success other than simply counting visitors. The institutions use ad hoc observational techniques in the museums. They also gauge reactions through published reviews in magazines and periodicals.
The usability walk-throughs were run in 12 sessions with a total of 70 participants who were screened for cultural interest and basic familiarity with the web. Subjects were first shown excerpts of existing web sites related to culture. Groups of approximately six subjects each were assembled in the TJ Watson Usability Lab in Hawthorne, NY. As much as possible, the groups were composed of 50% women, 50% men, and represented the diversity in income, education, background, race, and ethnicity in the tri-state area.
Each group was shown sections of two existing cultural web sites (children were shown only one site), followed by two new design ideas for the proposed site. After each demo, a facilitator led a group discussion of their opinions of the content in the demo. Then participants recorded their individual thoughts about the demo content (children had no written part). Then the next demo was shown. At the end of the adult session, participants were asked to answer several summary questions (children did not do this part). Over the course of all the walkthroughs, five existing web sites were shown and five new design ideas were presented.
In the first part of the session, we showed existing web sites where users could navigate through 3D replicas of exhibitions, interact with panoramic images, visually explore paintings and 3D sculptures, write and publish stories about the content, and obtain information about different layers of a painting.
The mockups of our new design ideas shown in the second part of the usability walk-throughs included five different possibilities for exploring cultural content:
Usability walkthrough results
A major finding of the usability walk-throughs was that most of the participants did not express interest in web sites that involved active interaction with the content or other people, such as when using a filtering system, creating a notebook, or chatting. The multimedia prototype was clearly the best received among the design ideas. Among the existing web sites, there was a preference for sites where the user was guided through an experience or discovery process, and in this case, participants strongly suggested the replacement of text by audio.
We summarize those findings by hypothesizing that in this kind of entertaining web experience, users want less clicking, more watching. Users seem to be very comfortable with the idea of a streaming web experience that leads them through artistic and cultural artefacts where, unlike television, the stream can be paused, replayed, or interrupted for further exploration. In fact, we found a strong desire for availability of additional information through hypermedia links and in-depth analysis of the works of art. Interestingly, some of the participants viewed the more interactive design concepts and existing web sites as work, not entertainment.
The idea of more passive experiences was further reinforced by many complaints about the difficulty of 3D interfaces, the excess of textual information, and the amount of work required to find information. The streaming multimedia concept seems to address this major concern by neither requiring the user to learn and exercise a large set of new skills nor requiring a large amount of attention and decision-making.
The usability walk-through also detected a desire for a "human voice" behind the multimedia experience; that is, a personal viewpoint in the exposition of the content. However, this voice needs to have the authority of an expert in the field or the human interest of a celebrity. In fact, a major complaint identified with chat systems was precisely the lack of trust and perceived value in the information provided by generic chatters.
We explicitly asked the usability walkthrough participants what factors would be important in their decision to re-visit the site on multiple occasions. The most frequent responses to this question were:
The modal response to the question "How often should the site be updated with new material" was one month, but there was considerable variance in the data.
Web site surveys
The web surveys were conducted by using a questionnaire that automatically popped up, in a separate window, whenever users accessed the home page of two partner art museums. Answering the 17 questions was optional, but throughout the month the survey was conducted, 830 and 1417 users answered the survey on each of the museum web sites. The survey was centered on questions about how the respondents use the web to obtain art and culture-related information and entertainment.
Web site survey results
About 81% of the surveyed visitors were visiting the web sites for the first time. About half of them (45%) got to the site through search engines and 17% through links from related sites. An estimated 10% came directly from other web pages.
About 25% were researching information on specific content, while 32% were interested in specific information about the museum (schedules, access, etc.). About 12% were looking for fun-like activities and only 2.4% declared interest specifically in shopping.
The absolute majority of the visitors (90%) were alone while visiting the sites. The most important aspects in the visit were the "content" (46%) and the ability to pre-visit the museum (31%). Only 14% ranked "usability" as their major concern.
About half of the visitors considered the sites "interesting", 21% saw them as "educational", and only 4% thought of the sites as "entertaining". This, in some ways, correlates with the objectives of the visitors as described above.
Visitors considered actually getting the information they wanted as the most engaging part of the experience of the site (39%). Similarly, when asked what part of the experience could not be replicated, 50% of the visitors mentioned the facility of getting information, and 35% mentioned the convenience of visiting the museum from home. The biggest frustrations with the web sites were
There was a slight predominance of males (58%) among visitors to the site. The age distribution was quite homogeneous, with a strong participation of people in their forties (22%) and fifties (11%).
Most of the visitors (58%) belonged to families having more than 5 hours/week of web usage and only 22% used the web less than 2 hours/week. As for connection time, 76% had a connection equal to, or faster than, 56Kbps, and 99% have a 28.8Kbps or better connection available.
To the users who answered the survey, the most popular education web sites were Discovery (6%), Yahoo, PBS, and the New York Times. The most popular entertainment web sites were Yahoo (6%), the Internet movie database, ESPN, and the onion.
Brick and mortar visitor surveys
A total of 99 visitor surveys were conducted at two of our partner institutions. The surveys at each institution were conducted on multiple days in order to get information from weekend and weekday visitors, as well as to survey individuals who might be visiting to attend a special event at the institution.
Brick and mortar visitor survey results
People enjoy integrating cultural experiences into their activities in order to create a rewarding social, entertainment, and educational experience - also, visitors to the institution are interested in learning and enriching themselves and their friends and families.
Many people selected the institution they were visiting because it provided an event that would satisfy both social and educational goals. A visit could also be integrated with other social events (visitors appeared to schedule a trip to the institution along with other activities such as a walk in the park, shopping, and dining). A majority of visitors visited to enrich themselves by viewing exhibits and special events. Many visitors indicated that the use of different presentation techniques (e.g., kiosks, films, human guides, lighting effects, etc.) greatly enhanced the learning experience.
The nature of the content dictates whether people are more focused on entertainment than education, and is primarily responsible for shaping the experience. As for the design of the web site, these surveys point out that there should be different treatments and different goals for different kinds of content. For instance, scientific content asks for an experience designed to be more educational than entertaining, while content related to comedy and drama should be entertaining and emotionally engaging.
There is a significant difference between the normal visitors and the audience for special events. Special events are less "social" than regular visits. In terms of the design of the site, this data should inform the number of community-related features (for instance, chatting) available for regular browsing visitors versus the audience for special events.
Conclusions of the Discovery Phase UCD activities
From the responses of the web surveys, we found that only a minority of people (4 to 5%) label the experience they have at museum web sites as "entertaining". In many ways, the participants in our research seem to lean towards defining an entertaining web experience as something closer to traditional TV, but enriched by the opportunity to explore and find extra information. However, our research also indicated that to access the majority of our targeted audience today, such web experiences must be available to users sporting modems of 56Kbits/second or more.
Based on the results of the discovery phase, we developed a design concept for the cultural web site based on the idea of multimedia tours guided by experts and artists. In our design, a tour presents information to the user continuously, from beginning to end, unless the user signals a desire to explore extra material or to exercise control. During the tour, opportunities to obtain additional information are presented as hot spots on the screen. All of the extra information is also available for exploration at the end of the tour.
The proposed web site consists of a collection of tours covering different areas of art and culture, such as painting, music, theater, and sculpture. In order to satisfy a large number of users, it seems to be necessary to provide a large number of tours and to add new ones regularly. Some of the design decisions presented below are, therefore, the result of the need to reduce production costs.
To cope with the requirement of maximum 56K bandwidth, we decided to explore multimedia experiences primarily based on still pictures and sound, instead of low quality video. At 56Kbits/second, video tends to become small and blurry, while at that speed it is possible to download audio with reasonable quality and reasonably good-looking pictures. Dealing mainly with stills also serves to reduce production costs, since shooting video is normally quite a bit more expensive than just audio recording and picture taking.
In our design, the main multimedia experience, or simply main tour, is enriched by the addition of user controls such as pause/resume/rewind/forward and by the inclusion of hot spots for two kinds of extra content: side tours and branches. A side tour is an extra, self-contained segment of multimedia, normally focusing on more specific aspects of the tour subject, which can be "inserted" into the main tour. A branch is a static web page with text, pictures, and links to extra information on a specific subject. The decision to create a side tour instead of a branch is based on the expected reach of the content. Since side tours are much more costly to produce than branches, we tend to produce side tours only for highly desirable extra information.
Figure 1 shows a snapshot of a tour. Most of the area is used by the content being presented by the tour (pictures, text, occasionally very short segments of video). On the bottom left side, a pictorial map gives the user a basic idea of the duration of the different scenes of the tour and the elapsed time. Rolling the mouse over the map brings extra textual information about each scene, while clicking on the picture of a scene interrupts the current scene and immediately starts the scene corresponding to the clicked image.
Figure 1 Typical scene of a tour with its navigation map, including links to a side tour and two branches
As the tour progresses, hot spots indicating side tours and branches appear on the screen. When a side tour is selected, the main tour is interrupted and the side tour is played. When the side tour finishes, the main tour automatically resumes from the point where it was left. A click on a branch pauses the tour and opens a new window on the browser, displaying the web page associated with the branch. To resume the playing of the main tour, the user is required to click on the pause/resume marker above the map.
When the tour finishes, all the content - main tour, side tours, branches - of the tour becomes easily available on the explore page. Figure 2 depicts the explore page associated with the tour offig. 1. Clicking on the tour map restarts the tour from the beginning of the scene that was clicked. Similarly, clicking on side tours and branches immediately starts them. The explore page is automatically presented to the user at the end of the main tour. However, during any moment of the main tour, the user can access the explore page by clicking on the corresponding hot spot on the right of the map (fig. 1).
Figure 2 Exploration page that allows access to the main tour and all side tours and branches.
The Two Pilot Tours
The design concept described above instructed the development of two pilot tours that are the main instruments used in the evaluation methods later described by Pinhanez and Karat (Pinhanez 2001, Karat C. 2001, Pinhanez 2001).
The first tour features the work of a toy pianist, Margaret Leng Tan. In the tour, the pianist talks about her involvement with toy pianos, how music is arranged for a toy piano, and her connections to Schröeder, the famous cartoon character created by Charles Schulz. Two side tours describe the history and functioning of toy pianos and the work of Margaret Leng Tan before becoming a toy pianist. The main tour lasts 4m15s, and the side tours take 1m18s and 50s, respectively. The tour also includes five branches.
The second tour talks about Ludwig van Beethoven and his Ninth Symphony. In this tour, a narrator leads the user through aspects of Beethoven's life and work, with comments also by a maestro and a concert pianist. Three side tours are provided, one about Beethoven's deafness and two others that let the user explore Beethoven's scores and his famous Heiligenstaden testament. Beethoven's main tour lasts 10m10s and the first side tour 2m00s. The other side tours, since they incorporate interactive elements, have no fixed duration, although their exploration usually takes around 60s each. There are also 5 branches available for user exploration.
After completing the development of the prototypes, we again engaged a sample of our target user population in a usability study to assess the success and value of our designs. The prototypes were tested with men and women from 21-55 years old. Test results indicate high user satisfaction with the tours. Users interacted relatively infrequently with the tours, and the less they interacted, the more they reported feeling engaged and entertained by the experience. This supports our initial hypothesis that the tours are most entertaining when they are experienced as they were designed to be; that is, watched as a TV-like experience. The results of the usability study give support to the contrarian "less clicking, more watching" design approach identified in the discovery phase.
Alberg, C. & Shneiderman, B. (1994). Visual information seeking: Tight coupling of dynamic query filters with starfield displays. Proceedings of CHI '94 (Boston MA, April 1994), ACM Press, 313-317.
Karat, C. et al. (2001), Less Clicking, More Watching: Results of the Iterative Design and Evaluation of Entertaining Web Experiences. Submitted to Interact'01.
Pinhanez, C. et al. (2001), "Can Web Entertainment Be Passive", Submitted to International World Wide Web Conference, Hong Kong,
Pinhanez, C. et al. (2001), "Less Clicking, More Watching": An Option for Entertainment on the Web?" Submitted to CHI'01, Seattle.
Stone, M., Fishkin, K., & Bier, E. (1994). The movable filter as a user interface tool. . Proceedings of CHI '94 (Boston MA, April 1994), ACM Press, 306-312.
Viegas and Donath (1999), Chat Circles Proceedings of CHI '99 ACM Press.