Join our Mailing List.
Published: March 1999.
"Web Musing": Evaluating Museums on the Web from Learning Theory to MethodologyLynne Teather and Kelly Wilhelm, University of Toronto, Canada
Table of Contents
IntroductionIn the two years since the first Museums and the Web Conference in Los Angeles in 1997, many museums, both in North America and beyond, have launched onto the Internet. Museum professionals working with the Web have developed some discourse around the philosophy, purpose, management and evaluation of museum web sites that reaches beyond the particulars of technology and production. Attempts have been made to draw out questions of representation, authority and power (Walsh, 1997; Witcomb, 1997; Worden, 1997), and emerging issues around new museology (Teather, 1998).
At the same time, museum professionals are exploring the implications of the premises of the museum experience, meaning-making and the visitor-centred philosophy of constructivist education for the museum on the Web. Some efforts have been made to push the museum web experience into the territory of ecomusology and new museology, and to both theorize and create examples of participatory and collaborative museum Web environments (Teather, 1998; Friedlander, 1998). However, the theory and practice of evaluation and usability studies necessary to the development of museums' use of new media remain underdeveloped. While a number of studies have been produced, few are linked to emerging ideas of the museum visitor experience at the heart of the current museology.
Premises of "Web Musing"In this paper, the authors will look to two questions. First, what might museums accomplish on the Web? What is it of the museum experience that we wish to convey via the Web? Looking to both exiting practice and possibilities, we will discuss museum Web work in general, and examine the specific evaluation study of the University of Toronto AMICO Testbed Project. Examples of an expanded notion of the museum experience that moves to the local, the plural, and the participatory exist both in museology and on the Web, and these extend to address "human communities in all their vital diversity" (Menzies, 1996). What are the implications of such premises of participatory museology for the museum Web experience, and to a project like the University of Toronto AMICO Testbed?
The second question is: How do we know when we've accomplished our objectives for museum presence on the Web? We will explore methods of user study available to the museum Web worker through the adaptation of evaluation study in museums and by reviewing the human/computer interface field of New Media. Again, in looking to the example of the University of Toronto AMICO Testbed Project, we will discuss evaluation choices for museum new media projects and their implications for our purposes on the Web. The premise that any critical discourse on museums and new media must be based upon voices beyond the builders of the technology is central to this push for evaluation work around the Web experience of the Museum.
1. What might museums accomplish on the Web?Teather (1998) asked the following central questions in her exploratory paper for last year's Museums and the Web conference: Should the museum be transferred to the Web? Can the museum be transferred to the Web? What is it of the museum experience that we wish to convey via the Web, and in what manner?
A number of purposes or goals for museum presence on the Web have been identified up to this point. James (1997) conducted a survey of 33 museums to determine, among other factors, the "general purpose" of their web site. The responses are listed below in the order of most often to least often cited:
In comparison, Moritz (1996) pinpoints the following reasons why museums are making their presence felt on the Web in his brief framework of the objectives of museum web sites:
A virtual museum that takes its context from a real museum is embedded in the ongoing functions of that museum. The traditional museum is a complex of interlocking activities, spaces and functionality's [sic] (exhibits, education, research, excavation, marketing, publication, and outreach). The virtual museum can pick up these functions too. However, it can also extend the breadth of information on offer to present all collections and, allow multiple interpretations and perspectives. (Kenderdine, 1996)Kenderdine's point is an important one: museums can choose to replicate their existing functions via the Web, or they can enhance these functions and extend to offer new ones facilitated by the new medium.
Many museums, including the AMICO member institutions, are committed to increasing public accessibility to information about their collections of varying depths via internet databases available to a broadly or narrowly defined group. Museums have taken a variety of approaches to achieving this goal. Projects such as the Getty's MESL project (http://www.gii.getty.edu/mesl ) or AMICO (http://www.amico.net) have adopted educational site licensing as a solution, while the Fine Art Museum of San Francisco's Thinker is a good example of a site that offers public access to 70,000 of the museum's art images (http://www.thinker.org).
While the public access issue is increasingly addressed in new and effective ways, a literature review of current practice and issues, however, illustrates that few museums are providing Kenderdine's second museum function on the Web: a forum for multiple interpretations and perspectives.
The degree to which the representation of the museum on the Web should resemble its physical presence or "real" presence is still a matter of debate. (See papers on "Rich Experiences" and "Virtualizing Museums: The De-Materialization of the Institution" from Museums and the Web 1998 at http://www.archimuse.com/mw98/frame_speakers.html). Thinkers from within the museum field are beginning to conceptualize selective roles for technology that bring the best of museum functions to bear in the Web environment (Semper, 1998; Dietz, 1998). Despite this developing discourse, some key questions concerning the nature of the experience provided by museums on the Web remain:
[T]here are many questions left as to how museums can create constructivist learning opportunities that value the aspects of personal meaning making and how to translate this to the web effectively. We may also need to examine the degree to which constructivism fits the museum model. Clearly at this point the new discourse around museology aligns quite well with constructivist interpretations of the museum and museum experience and possibly for the web. (Teather, 1998)One key challenge for museums on the Web, then, is to exploit the potential of the Web environment, and to design web sites that facilitate a unique virtual visitor experience that cannot be duplicated in the physical museum. Through this recognition, museums can begin to design Web environments that provide a meaningful, or constructivist, learning experience. Suggestions of the fulfilment of the constructivist ideas and notions of meaning making in the museum Web experience exist within a number of sources. Sabin (1997) and others acknowledge that such a broad view of the potential of museum presence on the web allows for a deep experience for the visitor. This experience is rightly tied to the existing functions of museums, but expands their communication potential through the Web medium:
If we view the Web as a place where we can challenge the boundaries of traditional museum exhibitions, a place where we can experiment with interpretation rather than relying solely on images of objects and details of accessions, perhaps there is a way that all museums can benefit. Museum curators often have a strong and highly evolved narrative sense of their subject, and it is from this narrative richness that objects acquire much of their meaning. (Sabin 14)David Bearman raised the questions early along in the discussion of the visitor experience via the Web:
How can we make the facts of these objects sing to the virtual visitor? How can we enable them to have an experience? The first requirement for museums is to recognise that the networked environment is interactive, and therefore can be user driven. It enables us to respond to the visitor rather than pump information at him. If used to its best purposes, the networked environment enables a user to construct an experience with personal meaning. (Bearman, 1995)For Bearman and many other museum Web enthusiasts, the answers lie in the world of informatics, in information architecture that can provide a context that facilitates the users' understanding of information. Thesauri, schema, organic constructs - the terms of information architecture - must be addressed, but they must be blended into the user interface to reflect museological issues. The communication of restructural metadata that allows for user understanding of context, method and structure, is of particular importance to usability (Teather 1998).
Another group has raised deep questions of the nature of the visitor experience in the museum, and, by extension, in the museum on the Web. Museum educators are extending the frame of meaning in the "real" and virtual museum. In her book From Knowledge to Narrative: Educators and the Changing Museum, Roberts (1997) captures the evolution of thinking around "education" in museums and the emergence of ideas about the museum experience:
Education is not just about museums teaching visitors; it is about visitors using museums in ways that are personally significant to them. It is about the mismatch that arises between a museum's culture and a visitor's culture; and it is about negotiating that mismatch in a way that is respectful of both. The essence of the education enterprise is thus the making of meaning. Whether it involves visitors interpreting their experience or museum personnel interpreting collections, meaning making is at the heart of the endeavours of both. (Roberts, 1997; 132)For Roberts, education exists both in the making of meaning and within the "observing, comparing and evaluating [of] possible versions of the world:" tensions and conflict among views of the world brings a reconstruction of worldviews. Museums are shifting from a "Logico-sientific" paradigm, one in which a single, expert curator-scientist-driven version of the world is presented, to a "narrative" paradigm, where the multiple, visitor-driven versions replace the objective "truth" with "meaning" (Roberts 1997; 134). Roberts admits that our understanding of learning in museums remains limited. Only a few studies have explored the motivations and forces that inform visitors' experiences and perceptions of museums. Falk and Dierking's (1992) work suggests an "interactive experience model," of the social, the physical, and the personal, and offers a holistic understanding of the visitor in the museum, whether the experience is rooted in social interaction, reminiscence, fantasy, personal involvement or restoration. Silverman (1991; 1995) builds her work on meaning making on the interaction of memory and the present context. Memory consists of the individual's mental frameworks that include knowledge, expectations and norms, and of life events and situations. The present situ forms the visitor's recollections and connections, which in turn emerge out of self-identity, companions, and leisure motivations. For Silverman, the meaning that visitors make in the museum is shaped from two human drives, individuality and community.
Morrissey and Worts (1998) apply an understanding of meaning-making and the visitor experience to the museum on the Web. They begin at first principles in their consideration of the pre-conditions for experiments with museum interpretation and communication via the Web. They assert that museums must start from a redefinition of their roles in society if they are to make the best possible use of the potential of technology. Discussions of the use of technology in museums must therefore begin with the identification of who we are and who we want to be as museums within our multiple and varied identities. Morrissey and Worts warn against museums providing more information to more people, and instead advocate uses of technology that facilitate meaning making:
The real challenge is in better understanding how technology can illuminate and enhance the complex relationships between people and objects…At its best, technology can facilitate experiences in which visitors can both transcend and live more fully in their daily lives, thoughts, and activities. It can challenge visitors to reconsider or create new meanings. It can help visitors see their experiences in a context that connects them to other people, places, and times. It can help museums realize their institutional potential as they help people realize their individual potential. The final challenge then is no less than that of placing technology in the service of understanding and enhancing the human experience. (Morrissey and Worts, 1998; 170)
2. How do we know when we've accomplished it?The way in which we approach the second question is dependent wholly on the way we answer the first. The way in which we define the purpose of and audience for museum web sites determines, or perhaps should determine but doesn't always, the ways in which they are evaluated. This approach is consistent with the widespread evaluation of museum exhibitions according to specific objectives articulated by an exhibition team: the objectives set determine the way in which evaluation is used to investigate if the objectives are being met. An exhibit that is designed to meet specific educational objectives around a topic area will theoretically be evaluated with different tools than an exhibit that is designed to evoke sensation or to be experiential. Most exhibitions are designed to meet a variety of objectives, so a combination of evaluation techniques is the most appropriate method; so, too, with museums and the Web.
Museums have a long history of evaluation approaches that most recently come together under the heading visitor studies. The professional organization for museum visitor studies is the Visitor Studies Association (VSA). Their web site can be viewed at http://museum.cl.msu.edu/VSA/ The VSA offers conferences and publications that disseminate the widespread practice of evaluation in museums and other visitor-centred institutions, and, through this dialogue, shares current notions of best practice. While the evaluation of exhibitions and educational programs is widespread, these techniques have only seldom been extended to a consideration of the museum on the web (Campbell and Wells, 1996; Perry and Saunders, 1996). Museum Web evaluation remains a new direction for this growing field.
In her 1998 paper, Teather began to discuss the ways in which the practice of visitor studies can be adapted to the Web. She identifies a relationship between the goals of museums on the Web and the ways in which they are evaluated:
Piacente's typology of web pages is very useful particularly because it parallels the typologies we have developed for exhibition analyses. It also admits that there are multiple purposes not only to museums but to their web pages. Perhaps we should be developing some sense of the complex of museum web page types and that there may be differing purposes and evaluation criteria for them. (Teather, 1998)We can use the typology provided by Piacente, or that provided by Moritz, as a basis for first discussion here. These three-part frameworks for the purpose of museum web sites can be simplified and translated into two broad roles or functions of museums, and, by extension, of museum web sites:
With these two broad notions of the museum as guidelines for the function of the museum on the Web, we can begin to discuss the ways in which we evaluate two types of museum Web presence.
3. Evaluating Object and Information Oriented Museum Web SitesIf the primary function of the museum on the Web is to provide information and or images of objects in the collections, to encourage research and to communicate information, then evaluation will focus on the ways in which this transfer of information occurs. One of the most widespread ways in which museums fulfill this role is through the provision of collections databases on the Web. The literature reveals that those in the museum field tend to evaluate the following aspects of the delivery of collections databases:
This way of conceptualizing usability evaluation mirrors Teather's (1998) discussion of usability and design for HCI in her MW98 paper, in which she borrows a two-statement summary of the goal of user-centred design from Baeker et al. (1995):
4. University of Toronto AMICO Testbed Project ExampleThe example that will be used to examine the evaluation of object and information oriented museum web sites is the University of Toronto AMICO Testbed. Professor Lynne Teather of the Museum Studies Program at the University of Toronto is Project Principal for the Testbed, and Kelly Wilhelm is the Project Manager. Through work with the Art Museum Image Consortium, we have been investigating the development and implementation of evaluation methodologies for museum Web databases from a museological perspective.
The Art Museum Image Consortium (AMICO) is a not for profit association of institutions with collections of art that have come together to enable educational use of the digital documentation of their collections. Together, AMICO Members are building a joint digital library documenting their collections. This Library will ultimately be available to university, public library and Kindergarten through grade 12 educational communities. The major project of 1998/99 is the University Testbed in which twenty university campuses are involved in studying the use of a beta version of the AMICO Library in order to propose ways to improve the first public release of the Library scheduled for the 1999/2000 academic year.
The three University of Toronto campuses (St. George, Scarborough, and Mississauga), are participating in the Testbed. We are conducting research around the Library in order to explore the ways in which such a resource is used by the university community, and the ways in which it might be augmented for further use. As of January 1999, the methodologies have been developed, and implementation is pending in February and March. The final project report will be drafted by the end of April, 1999.
Part of the project in its initial phases was a research and literature review on which the background for this paper and for the evaluation process is based. A bibliography on Museums and New Media, a product of this phase of work, will be made available on the Web in February 1999 at: http://18.104.22.168/ccm/CF_Test/MusTech_Proj/
The purpose of the AMICO Library is "to enable educational use of the digital multimedia documentation of their collections" (AMICO FAQ's, 1999). Given this goal of educational access, the evaluation of the Library focuses on the effectiveness and efficiency of the distribution and delivery of multimedia museum information to designated users at the University of Toronto via the Web. The evaluation process consisted of the following phases:
Identify user groups:
For this project, the use of the AMICO Library was limited to "designated users" in the University of Toronto community. The primary user groups were faculty and students. Each group is relatively easy to identify and to obtain contact with, particularly compared with the undefined "public" audience that many unlicensed museum web sites reach. We are attempting to involve faculty and students from a number of faculties in the research.
Draft broad research questions based on AMICO objectives:
The primary objective of the University of Toronto AMICO Testbed research is to evaluate which technical, design and content characteristics of the AMICO Library are of most value to the university community. The research questions therefore address issues for the university infrastructure and for the museum experience in relation to an image database of museum objects: In what way(s) does the proposed AMICO Digital Library, in terms of technical, design and content characteristics, function as a resource for the variety of university users and how would it be augmented to facilitate use?
What are the implications of using an Art Museum-based digital image and information database in museum educational work delivered through the art museum, and, in turn, the art museum educational philosophy and practice to the AMICO database?
These questions were further broken down for each methodology as it was developed.
Research relevant examples of methodologies already used for similar purposes:
In this phase, we looked at examples from usability and library sciences, as well as museum examples such as MESL and other AMICO testbed participants. Many of these were shared in the discussion groups set up for the AMICO Testbed and available to the campus project teams via the Web.
Develop format and draft instruments:
A mix of quantitative and qualitative methodologies was developed: online questionnaire, questionnaire, participant observation, and focus groups. These different methodologies together allow for breadth and depth of results: the questionnaires provide feedback from a high number of participants on a few key efficiency and effectiveness issues, while the participant observation and focus group methodologies provide deep information from a smaller number of subjects.
Submit to colleagues for revision:
Project team members reviewed the methodologies for clarity and the strength and appropriateness of the methodologies.
Revise and submit to University of Toronto Ethics Board for approval:
This process is required for all University of Toronto research projects that use human subjects. Three faculty members with relevant research interests review all methodologies and procedures for ethical conduct toward all subjects: the process ensures confidentiality, rigour, and the least possible risk for subjects for the greatest possible benefit. While this was an important part of the process for the project, the approval took three months to obtain, and the seven-month project schedule has been structured for maximum efficiency as a result.
Pre-test instruments on a small population:
We are currently in the process of pre-testing the methodologies with a small group of subjects. This process will allow us to further hone all aspects of the process prior to wide-scale implementation.
Implement instruments with populations:
We are also in the process of contacting subjects to participate in the research. We have prepared for this process through the earlier stages of the project, in which we sent information about the database out to faculty and students and informed them of the research project.
Collect and analyze data and report to AMICO and project team at University of Toronto:
The project report will be completed by the end of April 1999, and the AMICO Testbed teams from the participant universities are invited to a meeting in June to share their results. Feedback from the reports and from the meeting will be incorporated in the subsequent iterations of the AMICO Library as it prepared for wide release for the 1999/2000 academic year.
The University of Toronto AMICO Testbed evaluation model has some unique strengths. First, the task of identifying and making contact with subjects was assisted by the restricted access of the AMICO Library, and by existing contacts between the researchers and the known university community. Obvious advantages exist in the total Testbed structure, in which multiple participant universities conduct research around the same product and a similar set of issues. Each Testbed participant university will, however, bring its own unique concerns and focus to the research, and these multiple emphases will feed into further development of the Library. An update on the preliminary research results of the University of Toronto AMICO Testbed project will be provided at the MW99 Conference in New Orleans in March 1999.
5. People and Meaning Oriented Museum Web Sites: Expanding the FrameThe AMICO Library is an example of the increasingly familiar object and information oriented approach that museums are taking to work on the Web. Its goals of the provision of multimedia museum information to educational publics are clearly articulated and can be evaluated using the methods described above. If museums are to move into the evaluation of people and meaning oriented museum web sites, however, a similar set of goals must be articulated against which evaluation can occur. The remaining sections of this paper begin to outline such goals, and to offer a framework of characteristics for such sites that can provide the basis for evaluation.
A number of museum web sites originate from the constructivist premises of visitors' individual experiences and meaning-making. These sites work to facilitate and encourage multiple voices and the exchange of stories both outside and inside the institution and between staff and visitors. If this is our task on the Web, museum meanings must be presented in a manner that focuses on visitors, and on their knowledge and meaning-making, which are in turn built on experiences and understandings. At the heart of this model of the museum Web experience is the presentation of museum information from a self-reflexive perspective that attempts to situate information in the context of the institution's nature, strengths and weaknesses. The message, then, is formed not as declaration but as an argument, and contains complexity and layers that invite visitors to weigh their own view against the museum's. This model moves from objective information to subjective knowledge, and from knowledge to meaning that is negotiated between the museum and the visitor.
How can the museum present the information on the Web in a manner that facilitates multiple voices and visitors' construction of meaning? How could the variety of ways of thinking about the visitor's experience be translated to the Web? What does such a model look like? Ideas about museum learning and meaning-making may provide such an approach to the museum Web experience. Museum Web professionals could take any of the variety of ways of thinking about learning - Kolb, Gardner, Myers and Briggs - and apply a model for the web experience (Hein 1998; 24). George Hein (1998) has synthesized some of this work from learning and educational theory, and this synthesis might assist in planning the museum meaning-making web site.
Hein has crystalized educational theories, each with its own epistemology and learning theory, into four domains that represent a typology of educational theory. He plots two continua, the vertical as the theory of knowledge and the horizontal, as a theory of learning. These aspects parallel the museum education experience; thus, by extension, they can encapsulate the museum Web experience.
In the top left quadrant lies the didactic, expository education frame of the traditional authoritarian "school" environment and of the "teacher" mode of communication. Museum web pages that emerge from the didactic approach are sequential and based on an intended order with a single correct path, beginning and end. Didactic components - in this case, web pages - describe what is to be learned from the web presentation. Such pages imply that what they are providing represents a single authoritarian "truth". The stimulus-response mode, rooted in early behaviourist work, is similar to the didactic, expository approach, except it makes no claim for the scientific or objective truth of the teaching. As Hein relates, few museums would admit that their educational practice is consistent with such an approach, even if this was the underlying message.
The right-hand side of the quadrant, discovery learning, shifts the focus from the teacher to the learner, or the visitor. Active learning involves the learner in hands-on, interactive experiences, and has been adopted as an approach by the children's museums, discovery galleries, and science centres. In the right top quadrant, however, Hein presents a particular type of discovery learning, one that arrives at a single "truth" through "learning by doing".
In the right bottom quadrant - constructivism - the learner is actively involved in both the way their mind is engaged and in determining the nature of knowledge. Social constructivists look to the setting outside the individual that impact on their reading of meaning: politics, economics and social phenomenon. Von Glaserfeld and the radical constructivists posit that knowledge exists only in the minds of the learner, so that the validity of the ideas is evaluated not by an objective truth but by their position within the constructed reality of the learner. In either case, the situation of the learner, whether defined within their singular mental functions or as affected by broader social phenomenon, drives a visitor-centered model of learning and experience.
The constructivist web site, then, would invite visitors to construct their own knowledge. It would employ a wide range of active learning approaches, present a wide range of points of view, and provide many entry points, with no specific path and no identified beginning and end. Such a site would enable visitors to connect with objects and ideas through a range of activities and experiences that employ their life experiences.
A few web sites have entered the quadrant of constructivism. A handful have extended beyond constructivism to foster visitor participation in the construction not just of the narratives about what they see, but in what they see, in the very nature of the museum not just of their experience. They invite the participant into the selection of object or event and in layering their meaning in with that provided by the site and by other visitors' records of their views.
L.A. Culture Net, (http://homelacn.org/LACN/) remains an important example of this type of museum experience on the web. Here, the emphasis is "people understanding and participating in their community through culture" through the Internet. The mode of relating to the content is participatory and collaborative among the cultural and educational organizations and individuals.
Castle Toller in Upper Austria (http://fgidec1.tuwien.acat/1002situations/), provides an inkling of the idea of visitor meaning-making within the frame of ecomuseology on the Internet. Two artists, Michele Kolnicker and Michael Kiselinger, created the Heimat Museum, a virtual museum, to gather ideas and images of "heimat", "home" or "homeland" in German, through the topics of childhood, community, food, language, living-spaces, surrounding, things, and school. Although the origins of the project was in the idea of preparing a display of web originated stories in the Castle for a regional festival in 1994, the web site still carries on as visitors to the site add stories, memories and images to reflect their idea of "heimat".
6. A Proposal for Layering the Museum Web Site According to Constructivist IdeasAs illustrated by the examples above, museum web sites that embody constructivist principles provide opportunities for their visitors to engage with museum or cultural content. If we are to push these discussions further, several layers of museum self-analysis can be employed to explore the meaning of museums themselves to their visitors. Three layers of analysis can be identified in the negotiation between the museum and the visitor:
As discussed above, theories of constructivist learning posit an active learner that creates his or her own personalized learning experience through interaction with the learning environment. The experience facilitated by this project will provide the "virtual visitor" with an opportunity to apply various lenses of interpretation to different types of museum evidence in order to understand subjectivity, interpretation and the functions of museums as communicators of meaning. Outcomes for the virtual visitor will include an increased appreciation for and understanding of Canadian museums and their collections. Project participants will learn and apply new interpretive and technological skills within the framework of emerging paradigms of our understanding of visitor's experience of the museum in person and on the web. The three parts of the project are illustrated in the diagram below.
In Part A, the Object Context, virtual visitors to the site will interact with a variety of types of museum evidence available online. The evidence types would ultimately include:
7. Evaluating People and Meaning Oriented Museum Web Sites: An Example ProjectIn November and December of 1998, we joined TrueSpectra Inc., a company that research, develops and markets servers and software for the delivery and manipulation of high quality images. The partnership was founded on the premise that we could combine TrueSpectra's technology with advanced ideas of museology and museum learning theory to create a demonstration of a potential application for museum/gallery and related institutions' usage. With the promise of this partnership, we had developed the capacity to move to product development and evaluation. Two essential ideological components were required for the study. First, the educational objective for the prototype would be consistent with constructivist theories and museum standards of learning adapted to the web environment. Second, a research/evaluation segment would be included that extended the evaluation and user study developed for the AMICO project from September 1998 through April 1999, into the context of this new project. The assumption was also that a prototype, however humble, could be the basis for the development of a project proposal for subsequent phases of research funded by partner agencies. With limitations of time and resources, we took Part A of the original proposal, the Object Context, and decided to develop some elements as a prototype and test them in a preliminary study. Under a six week project timeline, we decided to focus our selection of museum evidence to fine art, and obtained images from Toronto artist Laura Higgins for the prototype.
We will apply only a few of the lenses of experience - digital perception, physical information, curatorial, conservation, exploratory, experiential and participatory - to these objects as a demonstration of what can be accomplished with museological content and cutting edge technology.
At the time of writing (January 1999), we have planned six weeks of study and development. Evaluation is integrated into the project plan at three stages: the paper storyboard, the electronic template of the prototype, and by experts in the museum field upon its completion. The methodologies that we will use will expand the qualitative methods developed for the AMICO study; participant observation and focus groups. We hope to also conduct long interviews structured around a few key questions that will allow virtual visitors to explore their experience with the prototype on their own terms. A "talk aloud" methodology that focuses on the meaning of the activity to the visitor, rather than on the visitor's navigation through the site, will be considered as a part of this work. It is expected that the demo evaluation will provide an opportunity to test the effectiveness both of the demo itself and of a series of evaluation methodologies for people and meaning oriented museum web sites.
8. Conclusions and Next StepsWhile the current demo project allows us to work only with a small part of the envisioned web site, the partnership will set the context for subsequent phases of work. These phases, with the full proposal and all three parts, will allow us to hone the methodologies that we develop in the prototype. The two separate projects together offer a unique opportunity to place the work with the demo in light of the insights gained in the AMICO Testbed Project and evaluation, and vice versa. As museologists working in the Web environment, we can exploit such opportunities to thoughtfully define museum web sites in relation to one another in order to distinguish their objectives and the corresponding evaluative techniques. It is equally important that we use such opportunities to build capacity for web evaluation in the field. As professionals, we must remain reflective, so that our work in this area can build on a solid existing foundation toward a greater understanding of the virtual visitor's museum experience. At the same time, our increasing understanding of the museum on the web will continue to enrich the ways in which we explore the totality of the museum experience, both virtual and real.
AMICO (1998). http://www.amico.net/
AMICO FAQ’s (1998). http://www.amico.net/faq.shtml#library
Baeker, Ron, Jonathan Grudin, William A.S. Bruxton and Saul Greenberg, Eds. (1995). Readings in human-computer interaction: Toward the year 2000. 2nd ed. San Francisco: Morgan Kaufman Publishers, Inc.
Bearman, David (1995). Museum strategies for success on the internet. Paper presented at Museum collections and the information superhighway, Science Museum, London, (May 10, 1995). www.nmsi.ac.uk/infosh/bearman.htm
Campbell, Heather and Marcella Wells (1996). Assessment of museum world wide web home page formats. In Marcella Wells and Ross Loomis (Eds.) Visitor Studies: Theory Research and Practice Vol. 9. (pp. 216-25). Jacksonville, AL: Visitor Studies Association.
De Kerckhove, Derrick (1997). Connected intelligence: The arrival of the web society. Toronto: Somerville House Publishing.
Dietz, Steve (1998). Curating the web. Museums and the Web98 Papers. Archives & Museum Informatics, 1998. http://www.archimuse.com/mw98/papers/
Exploratorium (1998) http://www.exploratorium.edu/
Falk, John and Lynn Dierking (1992). The museum experience. Washington, DC: Whalesback Books.
Fine Art Museum of San Francisco (1998). The Thinker. http://www.thinker.org/
Friedlander, Larry (1998). Models for a new visitor-centered museum: Using the web to create community and continuity for the museum visitor. Museums and the Web98 Papers. Archives & Museum Informatics, 1998. http://www.archimuse.com/mw98/papers/
Garzotta, Franca, Maristella Matera and Paolo Paolini (1998). To use or not to use? Evaluating usability of museum web sites. Museums and the Web98 Papers. Archives & Museum Informatics, 1998.
James, Stephanie (1997). Survey results. Web site, last viewed February 2, 1998, last updated March 5, 1998. http://www.chass.utoronto.ca/~sjames/museum/survey.htm
Kenderdine, Sarah (1996). Diving into shipwrecks: aquanauts in cyberspace." Paper Presented at MCN/CHIN 96.
MESL (Museum Educational Site Licensing) (1998) http://www.ahip.getty.edu/mesl/
Menzies, Heather (1996). Whose brave new world? Toronto: Between the Lines.
Moritz, Tom (1996). World wide web development in the museum context: An overview. CHIN Heritage Forum Web Site, viewed 1996. http://www.chin.gc.ca/Resources/Forum/e_forum.html
Morrissey, Kristine and Douglas Worts (1998). A place for the muses? Negotiating the role of media in museums." In Selma Thomas and Ann Mintz (Eds.) The virtual and the real: media in the museum (pp. 147-72). Washington: AAM.
Perry, H. Elizabeth Stuart and Carol D. Saunders (1996). Integrating evaluation into a technology education program. In Marcella Wells and Ross Loomis (Eds.) Visitor Studies: Theory, Research and Practice Vol. 9. (pp. 247-55). Jacksonville, AL: Visitor Studies Association.
Piacente, Maria (1996). Surf’s up: Museums and the world wide web. University of Toronto Research Paper, Master of Museum Studies Program.
Roberts, Lisa (1997). From knowledge to narrative: Educators and the changing museum. Washington, DC: Smithsonian Institution Press.
Sabin, Richard (1997). Museums and their websites: An examination and assessment of how museums are coping with the challenge of the world wide web. Journal of Conservation and Museum Studies 2 (May 1997). http://palimpsest.stanford.edu/jcms/issue2/sabin.html
Semper, Rob (1998). Bringing authentic museum experience to the web. Museums and the Web98 Papers. Archives & Museum Informatics, 1998.
Silverman, Lois (1991). Of us and other ‘things’: The content and functions of talk by adult visitor pairs in an art and a history museum. Ph.D. dissertation, Indiana University, Indiana.
Silverman, Lois (1995). Visitor meaning-making in museums for a new age. Curator 38 (3), 161-70.
Teather, Lynne (1998). A museum is a museum is a museum… Or is it?: Exploring museology and the web. Museums and the Web98 Papers. Archives & Museum Informatics, 1998.