Overview of MW98: Why you should attend MW98 Learn new skills to enhance your museum site Explore issues and controversies facing Museums and the Web Experts featured at MW98 Commercial products and services to enhance your web site Organizations supporting MW98: Online interchange regarding the virtual museum experience Juried awards to best web sites in 5 categories Overview of MW98: Why you should attend MW98 Learn new skills to enhance your museum site Explore issues and controversies facing Museums and the Web Experts featured at MW98 Commercial products and services to enhance your web site Organizations supporting MW98: Online interchange regarding the virtual museum experience Juried awards to best web sites in 5 categories

Overview of MW98: Why you should attend MW98 Learn new skills to enhance your museum site Explore issues and controversies facing Museums and the Web Experts featured at MW98 Commercial products and services to enhance your web site Organizations supporting MW98: Online interchange regarding the virtual museum experience Juried awards to best web sites in 5 categories

Archives & Museum Informatics

info @ archimuse.com

www.archimuse.comArchives and Museum Informatics Home Page

published April 1998
updated Nov. 2010


To Use or Not to Use? Evaluating Usability of Museum Web Sites 

Franca Garzotto, Maristella Matera, and Paolo Paolini 

HOC-Hypermedia Open Center 

Department of Electronics and Information, Politecnico di Milano P.zza Leonardo da Vinci, 32 - 20133 - Milano, Italy 

E-mail: {garzotto, matera, paolini}@elet.polimi.it 

Phone: +39-2-23993623; Fax: +39-2-23993411 



1. Introduction 

2. The SUE (Systematic Usability Evaluation) Approach 
    2.1 What is SUE?  
    2.2 The SUE Preparatory Phase  
    2.3 The SUE Execution Phase: Inspection and User Testing 

3. The SUE Preparatory Phase for On-line and Off-line Hypermedia  
    3.1 The HDM Model  
    3.2 Usability Attributes 
    3.3 Abstract tasks 

4. Examples of Inspection Findings 
    4.1 Musei Vaticani 
    4.2 National Gallery of Art in Washington 
    4.3 Further Examples (Virtual Uffizi, and Kimbell Museum) 

5. Conclusions 

6. Bibliography 

1. Introduction  
The advent of the World Wide Web has introduced a new communication channel, through which people can easily access museum content, events, activities, and schedules. Since the use of museum web sites is spreading all over the world, and these sites are attracting an increasing number of "virtual" visitors, it has become crucial to improve their usability, i.e. the visitor's ability to use these sites and to access their content in the most effective way. As a consequence, it has become compelling to provide both quality criteria that WWW sites must satisfy in order to be usable, and systematic methods for evaluating such criteria. 

This paper describes a technique for evaluating the usability of hypermedia applications, either off-line (i.e., multimedia CD-ROMs or Information Points) and on-line (i.e., Web sites). The approach presented here is based upon a general methodology named SUE - Systematic Usability Evaluation. Four attributes can be used to characterize SUE: heuristic, empirical, systematic, and model-based. It is heuristic according to the definition proposed by Nielsen [Nie93], in that it identifies a set of usability principles, or heuristics, that evaluation experts must check when the application is inspected for usability. Heuristic evaluation falls into the broader range of usability engineering methods called "inspection", which do not involve end users, but only "expert" evaluators. SUE is also empirical, in the sense that user testing is used to validate and refine the result of the inspection. The combination of inspection and empirical testing ensures the most accurate evaluation results, coupled with cost-effectiveness. SUE is systematic in the way usability inspection and empirical testing are performed. The inspection is carried on by executing a set of predefined evaluation activities, called abstract tasks. The use of abstract tasks makes the heuristic evaluation better organized and more effective, and also encourages standardization across different evaluators and evaluation processes. The empirical testing is carried on by requiring some end-users to perform concrete tasks, i.e. specific activities, based upon the result of inspection. The use of concrete tasks makes the empirical testing more organized and cost-effective. SUE, finally, is model-based since models are used to precisely shape the evaluation activities. In this paper, as we are examining hypermedia, we have used HDM (Hypermedia Design Model), developed at Politecnico di Milano. 

The rest of this paper is organized as follows. Section 2 briefly presents the overall approach of the SUE methodology. In section 3, SUE is specialized for hypermedia and we will discuss the model, the usability criteria, and (a sample of) the abstract tasks, that represent the cornerstones of our method. Section 4 illustrates our approach by presenting some (fragments of) evaluations of museums sites. Section 5 draws some conclusions. 

2. The SUE (Systematic Usability Evaluation) Approach  

2.1 What is SUE? 
SUE is a general methodology for usability evaluation of interactive systems, developed at Politecnico di Milano, in co-operation with the industrial partner CORINTO (COnsorzio RIcerca Nazionale Tecnologia ad Oggetti - an Italian consortium involving IBM-SEMEA, APPLE-Italy, and SELFIN), and the Universities of Lecce and Bari. 

The aim of SUE is twofold: 

  • to support a broad, multi-dimension analysis of usability, and
  • to make usability evaluation systematic and cost effective.

Most existing approaches to usability evaluation address only general interface issues, e.g. the lay-out design, the choice of icons and fonts, the interaction style, etc.. The main assumption of SUE is that usability analysis, instead, should consider the specific nature of the application to be evaluated. 

SUE suggests analyzing a system from different perspectives: 

  • general perspective, where the features common to all interactive applications are considered;
  • application-category perspective, that considers only the peculiar features of a specific functional class of applications (e.g., "hypermedia", "word-processor", "transaction processing system");
  • application-destination perspective, that focuses on the specific domain of an application within a given category (e.g., "tourism" hypermedia, or "education" hypermedia).

For each different perspective considered during the evaluation, SUE requires two operational phases of evaluation: i.e. the preparatory phase, and the execution phase, briefly described in the rest of this section. 

2.2 The SUE Preparatory Phase 
The purpose of the preparatory phase is to create the conceptual framework that allows the evaluation to be carried on in a systematic and effective way. The preparatory phase (for each perspective) consists of three different activities: 

  • the choice of a model;
  • the definition of a set of usability attributes; and
  • the specification of a set of abstract tasks.

The model identifies unambiguously the constituents of an interactive system that are the "entities of interest" [Fen91] for evaluation. The term "model" is used here in a very broad and loose sense, intending a set of primitives capable of describing an application. SUE does not prescribe which model should be used; the only requirement is that a model is used, and that the chosen model is sufficiently expressive to describe, in a non-ambiguous way, all the "entities of interest".  

The usability attributes are quality factors (closely related to the specific evaluation perspective being considered), under which usability is decomposed, in order to be more specifically analyzed and measured.  

Abstract tasks are "generic" operational activities that evaluators must perform during inspection. We call them "abstract" since they are formulated independently from a specific application, and they do not refer to the specific objects and functionality of a specific system, but, rather, to classes of objects and functionality that are relevant for the chosen perspective.  

2.3 The SUE Execution Phase: Inspection and User Testing  
The execution phase occurs each time a specific product is evaluated, and systematically combines inspection and empirical testing.  

Inspection is a generic term for a range of usability engineering methods that have seen an increasing widespread use. As we mentioned in the introduction, inspection methods do not involve end users, but "expert" evaluators only; since they "save users" [Nie94], and require neither special equipment, nor lab facilities, they are cheaper than other evaluation techniques. Another defining characteristic of inspection methods is that they are more informal than other techniques, since evaluation is "based on the considered judgment of the inspectors" [Nie94]. If compared with other inspection methods, the novelty of SUE-based inspection is that evaluators systematically exploit the results of the preparatory phase, in order to be more organized, standardized and cost-effective.  

Inspection is carried out first, and detects a preliminary set of potential usability problems. It is then followed by empirical testing, which consists of selecting a sample from the real user community, assigning them a set of tasks to be performed by using the application, and recording their actions. The novelty of SUE is that empirical testing is precisely driven by the output of inspection.  

3. The SUE Preparatory Phase for On-line and Off-line Hypermedia  
Since the focus of this paper is not SUE, in general, but its application to hypermedia, this section presents the conceptual framework tailored for hypermedia: the chosen model, HDM, the set of usability attributes, and a sample set of abstract tasks.  

3.1 The HDM Model  
HDM defines a number of dimensions along which a hypermedia application can be modeled: structure, navigation, behavior, user control, and presentation.  

Structure describes how the application content is organized.  Navigation concerns the definition of the browsing paradigms, i.e., the actual links available to the user for exploring the application structures. 

Behavior concerns two aspects:  

  • the dynamics of time-based media such as video, sound, animation, i.e., how their state evolves with time, by effect of user interaction or in dependency of the state of other media;
  • the dynamics of links, i.e., the effects of link traversing on the state of time-based media in the source and destination nodes.

User control concerns the operations available to the user to interact with the application.  Presentation concerns how all the previous features are "shown" to readers, and is related to the lay-out of objects (anchors, buttons, windows, menus, etc.), and their visual properties.  

In its current version, HDM focuses mainly on structure, navigation, behavior, and user control. In this section we introduce a very short synthesis of the model, mainly addressing structural and navigational primitives, which are the most relevant for modeling WWW hypermedia. The reader is referred to [Gar93Gar95Gar98a] for a complete description of the HDM model.  

HDM distinguishes between hyperbase structures, which constitute the so called hyperbase layer (hyperbase for short) of the application, and access structures, which constitute the so called access layer. Hyperbase structures are used to represent domain information, while access structures provide entry points to the hyperbase.  

The hyperbase consists of typed entities and typed semantic connections. Semantic connections represent domain-dependent relationships among (parts of) entities. Entities denote conceptual or physical objects of the application domain and are composite representation structures; their logical constituents are called components, and are organized according to some topological patterns (e.g., sequences, trees, lattices). Components in turn are made of a set of typed nodes. Different nodes within a component describe different "perspectives" along which the component subject can be represented. Nodes are the containers of content; they correspond to "pages" (or, sometimes, page sections), and aggregate a number of content elements called slots. A slot can be complex, i.e., it may include a number of different media, but it is always an atomic unit from the user interaction point of view. A slot can be static or dynamic, depending whether it stores time-independent media (such as formatted data, text strings, images and graphics) or time-based media (as video, sound, or animation). The type of a slot is defined by the conceptual and physical nature of its media.  

The access layer consists of collections. A collection groups a number of "members", in order to make them accessible. Members of a collection are hyperbase elements (entities, components, or nodes) or other collections. Collections including other collections among their members are called nested. A collection may have a center node, (although it is not mandatory) which is the starting point of the navigation within the collection and usually contains some information about the collection itself. Members of collections are collected according to some semantic criteria (e.g., in a museum application, "all paintings of a specific painting School"), or according to an expected user's goal (e.g., "the top ten paintings" for a quick visit of the museum pieces). Guided tours [Tri88] or tables of contents, commonly found in hypermedia applications, are modeled by collections in HDM.  

Navigation defines how to explore entities and collections. It is specified by means of navigational links (links for short) that connect nodes within the various structures. Navigational links are classified as structural, applicative, or collection links. Structural links connect nodes within an entity according to its topology; applicative links connect nodes of different entities related by some semantic connections; collection links connect the constituents of a collection. Links in the various categories are typed. A collection is called guided tour if it has only links connecting the center to the first member, and the each member to another. A collection is called index if it has only links connecting the center to each member, and vice versa. A guided tour index is a collection which includes both sets of links.  

3.2 Usability Attributes  
Usability attributes are empirical qualities that we expect from a well designed and a well implemented application. To define our usability attributes, we have first considered two general and universally acknowledged principles - learnability and efficiency, and have decomposed them into a number of hypermedia specific sub-principles that represent our usability attributes.  

Learnability concerns the features of an interactive system that allow novice users to use it initially, and then to attain a maximum level of performance [Dix93], and is decomposed into two attributes - consistency and predictability. Efficiency concerns the features that support the successful achievement of the user's goals, with a high level of productivity, and can be described in terms of accessibility and orientation.  

The HDM model helps focus our analysis of general usability principles, to identify the various classes of features which should be considered for usability, and to organize our attributes. Since HDM does not support presentation modeling, our approach does not consider presentation features. However, with the relevant exception of anchors, most presentation features of a hypermedia are largely independent of the specific nature of this class of systems, and concern evaluation from the general perspective. Furthermore, we will not consider in this paper usability features related to the dynamics of a hypermedia application, i.e., behavior of (and user control of) links and active media. For example, we will not discuss the usability of mechanisms that allow users to manipulate the state of video, sound, animation, etc. (by means of commands, such as "start", "stop", "pause", "restart", "forward", "backward" etc.), or the effects of link traversing. These features are relevant for hypermedia which include a significant amount of active media, which is not the case of most WWW applications. The reader is referred to [Gar97Gar98bGar98c] for a discussion of all these aspects.  

Consistency means that conceptually similar elements are treated in a similar fashion, while conceptually different elements are treated differently. From a structural perspective, consistency prescribes that entities representing domain objects of the same class aggregate components of similar types, and with a similar topology, components of the same type aggregate nodes of the same type. Nodes representing similar components, or similar perspectives of a component, contain similar sets of slots. Analogous considerations can be made for the access layer. From a navigational perspective, consistency prescribes similar navigation patterns (i.e., similar sets of navigation links) within entities of the same type, within similar collections, and across entities related by semantic connections of the same nature.   

Predictability focuses on the user's ability to identify the meaning of a structure or foresee the results of an interaction, having seen a similar structure or operation in analogous situations. Predictability measures the user's ability to form a predictive model of how the content is organized and how the system behaves, and also to "learn" complex situations after a significant amount of time spent with the system. The main factors that contribute to predictability are regularity, collection ordering coherency, and user's knowledge conformance.  

Regularity governs the degree of similarity in structures (and behaviors and user control, not discussed here), not only in similar situations as it is prescribed for consistency, but also across the overall application. It can be measured in terms of use of recurrent topological and navigational patterns, both for entities and collections.  

Collection Ordering Coherency requires that the navigation order among the members of a collection corresponds to the order in which such members are described in the collection center (e.g., by means of a list of descriptors such as titles, icons, miniaturized pictures, or similar). If the center shows the descriptors of two members, say X and Y, one after the other, according to some visual order, navigating forward from X the user expects to find Y, and not something else.  

User's Knowledge Conformance measures how well the elements available in the application match the mental model of the intended users, i.e., their set of assumptions, information, or preconceptions about the application domain, about hypermedia in general, and about the actual system. Clearly, different users have different mental models, and measuring user's knowledge conformance of a system is a very subjective process. The evaluator should verify if structures, navigation, behavior and interaction metaphors available in the application are appropriate for representing domain concepts, semantic relationships, styles of use that can be considered somehow standard, or at least intuitive, in a given cultural context and for the intended class of users.  

Accessibility measures how easy is for the users to locate the piece(s) of information they are looking for. It promotes usability particularly for users who have a specific information goal or must perform a specific task. Accessibility means that, from a given starting point, users can quickly locate the items that are needed for their task, without navigating through non-relevant material; or, alternatively, they can quickly discover that those items are not in the application. A number of factors contribute to accessibility. Some of them concern the completeness and efficacy of access structures. Completeness refers to how the organization of access structures and the content of their centers can be mapped to the set of hyperbase entity types and to their instances. When users enter a WWW site, for example, the home page, i.e., the center of the top level index of the application, should immediately give a global sense of the application content, and the perception of the hyperbase entity types. If an entity type or its instances are not mentioned by any collections, neither at the top level, nor at some level of nesting, users might not discover them, unless they traverse some applicative link (if any) in the hyperbase. The efficacy of access structure organization is mainly related to the trade-off "breadth versus depth" in nested collections (i.e., the number of collection members, at a given level, versus the number of levels of sub-nesting). For efficiency, it is often more appropriate to minimize the levels of nesting. Another important factor for efficiency is the navigational richness, i.e., the abundance of links connecting the various information elements. Which links are more appropriate in which situation is largely dependent on the nature and complexity of content. In a short and simple-content guided tour, for example, the link "next" - from a member to the following one, and its inverse ("previous"), might be enough. Still, in a complex guided tour, made of several steps and storing complex content, users may also need to jump to the end or to the beginning of the tour, or to rapidly jump to any member starting from the current one.  

Orientation is related to the "getting lost in the hyperspace" problem, a common usability issue for large hypermedia. Orientation measures the users' ability to understand their current location and their own movements, to grasp their current navigation context, and to return to previously visited items. A number of aspects can be considered to support orientation, such as backtracking soundness, context observability, and reuse soundness.  

Backtracking is the mechanisms by which the user can return back to previously visited nodes, and is one of the most common orientation facilities in hypermedia, especially on the WWW (where it is provided directly from the browser). Unfortunately, applications very often provide the backtracking function in place of explicit navigation links (e.g., as a surrogate for inverse links), thus overburdening backtracking commands with a number of meanings.  

Context Observability concerns the possibility for users to evaluate the actual status of their navigation session, and to understand the positioning of the current node within larger structures or within the entire application. Many applications use active maps and overview diagrams, with indications of the user's current location (and of previous steps), both to summarize the application content and to support orientation. Observability can also be supported by means of some perceivable visual cues on the nodes. For example, the relative position in a guided tour, or in a linear sequence of nodes, can be presented by a descriptor, such as "3rd of 10" (which also helps users to perceive how long is the way to the last element).  

Reuse Soundness analyses if reuse is well designed. Reuse means that structures, links, and operations, already used somewhere, are used again in a different context and for different purposes. Reuse has a number of advantages but it may also have some critical drawbacks. To avoid the increase of cognitive complexity and disorientation, reused elements must not include any context-dependent feature, or they must be adapted to the new context [Gar96]. For example, if a piece of content in a node depends on a given context, it should be removed when reusing the node in another context (and may be replaced with information needed by the new situation). Alternatively, if there is important information that is left implicit in a node and can be induced from a given context, it should be made explicit when the node is reused in a new context.  

3.3 Abstract Tasks  
Abstract tasks describe the activities that, during the inspection, the evaluators must perform in order to detect potential usability problems. We use the term "abstract", since they are formulated independently from a particular application, and they refer to categories of application constituents.  

An abstract task is described by five elements: the identification code; the title; the focus of action, i.e., the constituents which are the focus of the activity; the activity description, which describes what evaluators have to do, the intent, i.e., a short statement explaining what is the rationale of the abstract task. Optionally a comment, clarifying some concepts or definitions mentioned in the previous elements, is provided. The terminology used for the formulation of abstract task is based on the HDM model.  

The reader is deferred to [Gar98a] for a complete list of thirty abstract tasks, which cover the complete set of features of an hypermedia application. In the rest of this section we shortly introduce a few examples, just to provide the reader with the "flavor" of what abstract tasks are. The abstract tasks described here are mainly concern with the primitives that in HDM are called access structures, i.e., the collection of objects upon which indexes and guided tours are built, that are commonly used in almost every WWW application. In particular the first three abstract tasks (from AS-O1 to AS-O3) refer to the organization of the access structures, while the last four (from AS-N1 to AS-N2) refer to the navigation within the access structures.  

Title: "Extent of access structures"  
Focus of Action: an entity type, and the set of collections.  
Activity Description: consider an entity type (e.g. "painting"):   
1. verify if there are collections (indexes or guided tours) which allow the direct access to its instances;  
2. verify if there is at least a collection which allows the access to all its instances.  
Intent: to check if access methods efficiently support access to the hyperbase entities.  
Comment: a typical usability problem occurs when some entity types, i.e., classes of application objects, are not directly accessible by any collection (index or guided tour), and the only way to access them is through links from other entities. Another type of problem arises when the collections do not cover the complete extension of the entity type, in the sense that some of the instances are left out. Both types of problems are, surprisingly, quite common.  

Title: "Information quality in collection centers"  
Focus of Action: a collection center (i.e., the front page of an index or a guided tour).  
Activity Description: verify if the information content in the collection center accurately describes the content of the collection itself. For example, verify:  
1. the completeness, i.e., if all the members of the collection are clearly identifiable;  
2. the correctness, i.e., if the description of each member precisely describes the member itself;  
3. the ordering, i.e., if the order in which members are listed precisely corresponds to the navigation order in which members are traversed in the collection.  
Intent: to verify how well the center of a collection supports user's understanding of what is and what is not in the collection.  
Comment: centers of collections, i.e. the page through which indexes or guided tours are presented, are crucial for the effective usability of application. Imprecise, incomplete, or low quality information in a center, incurs fruitless user navigation (in the sense that they access items not really relevant to them) or can result in potentially interesting pieces of information being overlooked.   

Title: "Nesting levels of access structures"  
Focus of Action: the set of collections (i.e. indexes and guided tours).  
Activity Description: starting from the home page, reach the hyperbase objects (entities), following different paths across indexes and guided tours.  
Intent: to identify the access structure depth, and verify if there are hierarchies of collections with a high degree of nesting.  
Comment: it is always a difficult trade-off to structure a large application in the sense of "breadth" or "depth". Sometimes nesting is a necessity, but it should be kept in mind that an high level of nesting almost inevitably creates usability problems (in the sense that the reader may never reach the bottom of the nesting, or find his/her way up from the bottom).  

Title: "Complexity of collection navigation patterns"  
Focus of Action: a collection, and the set of collection links.  
Activity Description: in a collection:  
1. verify the complexity of the collection navigation pattern. For example:  

  • from the collection center, access a generic collection member;
  • from a generic member of the collection, access:
    1. the previous and the next member in the collection order;
    2. the first collection member and the last one;
    3. another arbitrary member;
    4. the collection center (if any);
  • from the first (respectively, the last) member, try to go "previous" (respectively "next");

2. verify the appropriateness of such a pattern, in accordance with the collection content and structure.  
Intent: to verify if the navigation pattern of a collection is adequate for the content and the purpose of the collection itself.  
Comment: again it is a difficult trade-off between rich, complex navigation patterns, which can be difficult to handle, and simplified, easily understandable navigation patterns, that may lack the necessary power. This is an area where, despite the difficulty of finding a proper design choice, technical mistakes are easily found.  

Title: "Up-navigation in nested collections"  
Focus of Action: nested collections (indexes and/or guided tours).  
Activity Description: verify the existence and the appropriateness of mechanisms for navigating "upwards" within nested collections. For example, starting from the home, go down in the hierarchy of collections, and then:  
1. go up through the same indexes;  
2. go up through different indexes (if any);  
3. traverse applicative or structural links, then try to go up to the higher level indexes, up to the home.  
Intent: to assess if adequate navigation patterns are provided for nested collections. The relevance of this task clearly depends on the use (or abuse) of the nesting within the application.  
Comment: a typical problem that generally arises is that poor mechanisms are provided for navigating upwards through nested collections. Therefore, once users go down through the collection hierarchy, they do not have any way to go up to the centers of higher level indexes, and in particular to the home page.  

Title: "Visibility of navigation status in collection navigation"  
Focus of Action: a collection (index or guided tour).  
Activity Description: in a collection, access an arbitrary member, and identify its position in the collection structure, uniquely from what you see (forgetting the path that has been followed).  
Intent: to verify if members are provided with clear indications about their location in the collection.  
Comment: identifying the position of a member within a collection is a crucial aspect of user orientation. It becomes even more important (and technically more difficult) if the same member belongs to different collections or appears in different positions within the same collection.  

Title: "Backtracking versus collection navigation"  
Focus of Action: a collection.  
Activity Description: in a collection, verify if and how backtracking is used within collection navigation. For example:  
1. go to a generic intermediate member, without passing through the previous members in the collection order; then access the previous member;  
2. go to a generic member, without passing through the collection center; then go up to the center.  
Intent: to assess the effect of using backtracking commands as a surrogate for navigation commands.  
Comment: backtracking, which returns the users back to the previously visited node, is one of the most important and common navigation facilities. As discussed by Nielsen [Nie95], "almost all hypertext systems provide some form of backtracking, but not always very consistent". Many hypermedia (especially on the Web!) make an intense use of the backtracking function (oftentimes called "back" or, ambiguously, "previous") to overcome the lack of explicit navigation links. Although in some cases it may work, in most cases it does not, and it can result in user disorientation.  

4. Examples of Inspection Findings  
In this paper we only limit ourselves to the inspection activity. We have inspected a number of museum web sites, using our abstract tasks for hypermedia, a sample of which has been described in the previous section. The overall structure of the sites is quite simple (in the worst cases, just a tree structure), in comparison, for example, to structures that could be found in multimedia CD-ROM. Simple structure can be easy to manage, but ineffective, from a usability point of view. When more articulated and organized structures are attempted, some usability problems seem to arise. 


The examples listed below correspond to inspections done in the beginning of March 1998; some aspects of the sites may have changed in the meantime. Inspections, of course, have used the full range of abstract tasks, and not just the few sample described in the previous section. 

4.1 Musei Vaticani (http://www.christusrex.org/
Hosted within the "Christus Rex and Redemptor Mundi" site, the on-line guide to the Vatican Museum allows access to images of the works exhibited in the numerous museum galleries, from the Gallery of Tapestries to the Gallery of Paintings. 

A "work" (an entity-type, in the HDM terminology) is only described by a set of pictures (in most cases, just one). The access structure is organized in several collections, corresponding to the museum galleries. Each collection has a center, consisting of a scrollable list (sometime very long) of thumbnails, through which nice pictures of the art works can be found. 

This site is a good example of very simple structure, easy to grasp, with (relatively minor) usability problems: 

  • the lack of collection navigation across members ("works") requires a continuous navigation from the list of works (collection center), to a work, then back to the list to choose the next member, etc.. Some further mechanisms for navigating across members would significantly improve the usability of the site.
  • the positioning of a "work" in the context of a collection is, in general, unclear: there is no explicit indication of the collection the work belongs to, nor of its current position in the set of collection members.
  • at the bottom of each list of works (collection center), a navigation bar provides commands for going up to the overview of available collections, and for navigating to the center of the previous and next collections. This is a nice feature, but its positioning (at the end of long lists) is unfortunate: it is likely that many readers will miss it. Also for those who know that the navigation bar is there, it is annoying to have to scroll a long list to reach it.

4.2 National Gallery of Art in Washington (http://www.nga.gov 
In our opinion, this is one of the best organized and most enjoyable museum sites; it really offers a nice and usable preview of what is available (permanently or temporarily) at the National Gallery in Washington, a wonderful museum on its own. 

Also in this case the most important entity type is "work", describing a piece of the museum collection or exhibitions. The access layer includes a number of nested collections (indexes and guided tours), which allow access to the museum works according to different criteria (e.g., by work nature, by subject, by artist, by country, by historical period). A powerful search tool ("Search the Collection") provides an efficient access mechanism to the same contents.

A few results, from the inspections, are now described. Let us assume that the search facility is used in order to get information about the Italian Renaissance artist Sandro Botticelli. After reaching Botticelli's page (containing a short biographical note, and the list of the artist's works exhibited at the museum), we select one of the Botticelli's paintings, for example "The Adoration of the Magi". The corresponding page (shown in fig. NGA-1) includes a small picture of the painting, a caption, and an interesting text describing the work. Below the text, a set of anchors named "full screen image", "bibliography", "detail images", "location", "narratives", "provenance", "tour", "purchase reproduction", provide links to different pages related to this painting. The name Botticelli, on the side of the painting picture, is a link to the artist page. 

The page shown in fig. NGA-1, and the pages accessed by means of links "full screen image", "bibliography", "detail images", "narratives", "provenance", "purchase reproduction", store information which are intrinsically "part of" the current painting (and of no other paintings); these pages can therefore be regarded as components of the entity (of type work) "The Adoration of the Magi", and links to them are examples of structural links in HDM. Links named "location" and " tour", instead, can be used to access pages storing information which are not exclusive of "The Adoration of the Magi", being related also to other paintings. "Location", in fact, is an applicative link (in HDM terminology), which takes to the room where the painting is located, together with several works by various artists. "Tour" is a collection link . The first observation is that structural links are visually intermixed with links of different nature, i.e., with the applicative links and the collection links. This visualization may disorient the reader, since "location" and "tour" will lead him/her to a different object, with respect to the one currently active, i.e., "The Adoration of the Magi". Selecting the link "full screen image", the page shown in fig. NGA-2 is activated. From NGA-2 the link "information" leads to NGA-1.  

If from NGA-1 the link "bibliography" is selected, page NGA-3 is activated, with the usual set of links, but "bibliography" itself. From NGA-1, the link "detail images" leads to the page reported in fig. NGA-4, that is also accessible from page NGA-2; from page NGA-4, page NGA-2 is reachable, and vice versa. From NGA-1, the link "location" leads to the page reported in fig. NGA-5, which describes a room, and is completely outside the description of the painting. From  NGA-1, the link "narratives" leads to a page shown in fig. NGA-6. The page looks exactly as NGA-1, with the exception of the text: the first part of it is the same as that of NGA-1; the second part is new. First of all, the user may get confused by the fact that the text is initially the same: until the text is scrolled, the page looks exactly the same as NGA-1. This "self-reference" effect is increased by the fact that the set of links is the same; more surprising, selecting the link "narratives" from NGA-6, NGA-1 is the destination! 

Finally, since in page NGA-2 the link leading to NGA-1 is called "information", we think that the interplay between the link "narratives" and "information" is overall quite confusing. From NGA-1, "provenance" leads to page shown in fig. NGA-7, where the link "provenance" itself is missing. Let us now activate, again from NGA-1, "tour", which takes to the page shown in fig. NGA-8. This is the center of a collection (in the HDM terminology) that provides a guided tour of a few paintings. First of all, let us repeat that moving from a painting to a complete different context (a guided tour) should have been treated differently from the other (structural) links. Secondly, the anchor "back to 15th century Italian paintings" looks quite odd: the sense of "back" is unclear, since we have never been there so far. The explanation is simple: the navigation could have started by selecting "Collections" from the home page; from there, "Italian 15th Century" could have been selected, and then "Patrons and Artists in Late 15th Century in Florence", leading to NGA-8. Following this path, the label "back to 15th-century Italian paintings" makes sense, of course.

Whatever the path followed to reach the page of figure NGA-8, let us select the second painting: reaching the page shown in fig. NGA-9. If we compare figures NGA-1 and NGA-9, the small picture and its caption are the same. The text is different: in NGA-1 it is a general description of the painting, and in NGA-8 it focuses on the aspects of the painting that are relevant for the guided tour theme. More precisely, in NGA-9 it is exactly the second part of the text that we found in the component "narratives", shown in NGA-6. If the reader feels that the situation is a little confusing, we agree. 

The set of links from NGA-9 is slightly different, from those of NGA-1: "narratives", "detail images", and "tour" are omitted, and we wonder why. At the bottom of the page there are two new links: one allows the reader to proceed to the next step of the tour ("continue tour"), and the other ("back to gallery") enables the access to the tour center (and has therefore the same effect as the link "tour" in figure NGA-1). In this case too, if the reader feels that the situation is a little confusing, we agree. 

Navigating within the guided tour, the behavior is slightly different from that seen before: from the painting page, from the bibliography page and from the provenance page the tour can be continued (in the sense of moving to the next painting from figure NGA-8). From "full screen image" and "location", instead, the tour can not be continued (although the context is correctly retained). 

Our overall comment is that minor problems arise within the context of the painting and the context of the guided tour; major problems arise from the interplay between the two contexts. These are typical problems induced by the conceptual complexity and by the need to reuse the same material in different contexts. 

4.3 Further Examples 
Virtual Uffizi (http://www.arca.net/uffizi) is a private site about the "Galleria degli Uffizi" in Florence. We tried to retrieve information available about Sandro Botticelli (the same painter as the "The Adoration of the Magi"). This artist has many paintings at the "real" Uffizi Museum, and we expected him to be largely represented in the Virtual Uffizi too. We started our navigation choosing the "Artists Index", where artists are listed in alphabetical order. We could not find any entry for Botticelli under letter B! We tried with "Filipepi" (Sandro Filipepi is the real name of the painter), with no success. Finally, we discover that the alphabetical order is by first name and not by surname, and we finally discovered the way to access Botticelli under the letter S (Sandro, Filipepi called Botticelli!). The observation that this solution may disorient most readers seems quite obvious. 

In the very nice site of the Kimbell Museum in Forth Worth - Texas (http://www.kimbellart.org) the first sentence in the home page, reported in  fig. K-1, encourages the reader to start the visit from the most famous exhibits, including the famous "top painters". Each artist's name corresponds to a link to the artist's page. It is a bit tricky to discover how to access an artist not mentioned in the first sentence of the home page: although the access layer of this site also includes an Index of Artists, which intends to collects all artists in the Kimbell Museum, this index is not directly accessible from the home page, and it is not easy to realize how to find it. If we try to select "Gallery" from the home page, we get to a page with no information, but a link "Continue to Gallery Entrance", which finally leads to page where the link to the Index of Artists is available. If we go to the page of one of the top artists, we discover a link to the index there. If for example readers start by selecting one of the artists mentioned in the home page - e.g., Caravaggio, they access a long scrollable page describing this painter (fig. K-2a and K-2b). At the bottom of this page (fig. K-2b) the reader is encouraged to "Return to Artists Index" (the use of the term "return" is a bit disturbing, since the user might never have been there so far). If this link is activated, the reader reaches the center of the Index, i.e., the page partially shown in fig. K-3. Unfortunately, in the list of artists there is no entry named Caravaggio, nor Merisi Michelangelo, the real name of the painter. The Reader's surprise can be safely assumed. 

5. Conclusions 
Usability is important for museum web sites, as well as for all the other sites. A usable site allows the user to easily exploit the content of the site itself; a lack of usability, instead, may lead to user disorientation or, in the worst case, to deep frustration due to the inability to proceed in the exploration. 

Usability problems, in most cases, arise from an inconsistent design or errors in implementation. Even if design and implementation are well organized, (minor) usability problem may arise for a number of different reasons. Since it is impossible to make sure, in advance, that no problems exists, it is wise to plan, in any case, a careful evaluation. 

In this paper we have proposed a systematic and model-driven approach to the evaluation of usability. Our plan is to conduct now a set of experiments, where the findings of the inspection evaluation (done by experts) can be compared with the results of empirical testing done with samples of end-users. 

6. Bibliography  
[Dix93] Dix, A., Finlay, J., Abowd, G., Beale, R., "Human-Computer Interaction", Prentice Hall, 1993. 

[Fen91] Fenton N.E., "Software Metrics: A Rigorous Approach", Chapman & Hall, 1991 

[Gar93] Garzotto F., Paolini P., Schwabe D., "HDM - A Model Based Approach to Hypermedia Application Design" In ACM Trans. Inf. Syst., 11 (1), January 1993. 

[Gar95] Garzotto F., Mainetti L., Paolini P., "Hypermedia Design, Analysis, and Evaluation Issues". In Comm. ACM, 38 (8), August 1995. 

[Gar96] Garzotto F., L. Mainetti, P. Paolini, "Information Reuse in Hypermedia Applications". In Proc. HT'96 (Boston, MA), ACM Press, March 1996. 

[Gar97] F. Garzotto, M. Matera, P. Paolini. "SUE-based Inspection of Hypermedia". Technical Report 97-75, Dipartimento di Elettronica e Informazione, Politecnico di Milano, December 1997. 

[Gar98a] F. Garzotto, M. Matera, P. Paolini. "HDM'98: the Hypermedia Design Model Revisited", Technical Report 98-03, Dipartimento di Elettronica e Informazione, Politecnico di Milano, January 1998. 

[Gar98b] F. Garzotto, M. Matera. "A SUE-based Technique for Hypermedia Usbility Inspection". Accepted for publication on the International Journal New Review of Hypermedia and Multimedia, Taylor Graham Publisher. 

[Gar98c] F. Garzotto, M. Matera, P. Paolini. Model-based Heuristic Evaluation of Hypermedia Applications. To be printed on the Proc. of the International Working Conference AVI'98, L'Aquila, Italy, May 1998. 

[Nie93] Nielsen J., "Usability Engineering", Academic Press, New York, 1993. 

[Nie94] Nielsen J., Mack R. L., "Usability Inspection Methods", John Wiley & Sons, New York, 1994. 

[Nie95] Nielsen J., "Multimedia and Hypertext: the Internet and Beyond", Academic Press, London, 1995. 

[Tri88] Trigg R.H., "Guided Tours and Tabletops: Tools for Communicating in a Hypertext Environment". In ACM Trans. Inf. Syst.. 6 (4), 1988.   

Last modified: April 1, 1998. This file can be found below http://www.archimuse.com/mw98/ 
Send questions and comments to info@museumsandtheweb.com