Skip to main content

Museums and the Web

An annual conference exploring the social, cultural, design, technological, economic, and organizational issues of culture, science and heritage on-line.

Rethinking Evaluation Metrics in Light of Flickr Commons

Paula Bray, and Sebastian Chan, Powerhouse Museum, Australi; Joseph Dalton, New York Public Library, USA; Dianne Dietrich, Cornell University Library, USA; Effie Kapsalis, Smithsonian Institution Archives, USA; Michelle Springer and Helena Zinkham, Library of Congress, USA

Abstract

In the past several years, cultural heritage institutions, including archives, libraries, and museums, have been placing their collections in Web spaces designed for collaboration and communication. Flickr Commons is one example of a highly visible space where cultural heritage institutions have partnered with a popular social networking site to provide greater discovery to, access of, and opportunities to interact with image collections on a large scale. It is important to understand how to measure the impact of these kinds of projects. Traditional metrics, including visit counts, tell only part of the story: much more nuanced information is often found in comments, notes, tags, and other information contributed by the user community. This paper will examine how several institutions on Flickr Commons - the Library of Congress, the Powerhouse Museum, the Smithsonian, New York Public Library, and Cornell University Library - are navigating the concept of evaluation in an emerging arena where compelling statistics are often qualitative, difficult to gather, and ever-changing.

Keywords: Flickr, metrics, evaluation, crowdsourcing, statistics, images

Introduction

Flickr Commons launched in 2008 as a platform for enhancing the discoverability of cultural heritage collections and for allowing users to tag, annotate, and repurpose these materials. The Commons now boasts 46 members and tens of thousands of images with the copyright status, "no known copyright restrictions". The impact of the project has been monumental. Many of the institutions involved can boast hit counts on the Commons that far exceed those on their institution websites. While many of the numbers are certainly impressive, it can be argued that they are one of the least compelling statistics.

What, then, are the most meaningful statistics for a project that puts library, archives, and museum collections in a highly visible space outside the institution's traditional confines? Some of our standard and more established metrics, such as hit counts, only illuminate a small part of the story - the bigger picture includes the myriad ways users have interacted with our images, by adding notes and comments, by including them in their own curated galleries, by tweeting them, and referencing them on blogs. Since there are limitless possibilities for users to interact with digital materials, we should be developing metrics that reflect this landscape.

Developing new metrics for Web 2.0 collaboration can be difficult, however. Compelling evidence of active participation and response by an online user community is often qualitative, and evaluation of this feedback is often labor-intensive, as it is not easily automated, unlike quantitative metrics. User interactions are often confined to the originating platform, and so are highly scattered on the web. The images on Flickr Commons all carry the "no known copyright restrictions" usage guideline; some users reuse the images in creative non-commercial ways and many repost images on multiple alternative platforms. Finding evidence of interaction can be difficult, as there is no one tried-and-true strategy for pinpointing every user reference to a Flickr Commons image. Finally, the ways in which users can interact with these images is constantly evolving and changing; it is nearly impossible to anticipate all of the interactions that can take place.

This paper will examine how five institutions - the Library of Congress, Powerhouse Museum, the Smithsonian, New York Public Library, and Cornell University Library - are navigating the concept of evaluation in the Web 2.0 arena. While each institution has a different take on approaches to assessing the impact of a digital library collection that exists beyond its borders, common themes do emerge.

Institutional case studies

Library of Congress

The Flickr Commons initiative launched in January 2008 with the introduction of the Library of Congress account on Flickr (http://www.flickr.com/photos/library_of_congress). That first load featured two sets of popular images from the Library of Congress website: 1,600 color photos showing the Great Depression and World War II and 1,500+ images from the George Grantham Bain News Service. New photos are added each week. Today, over 12,000 images in 15 sets appear in the Library's Flickr account. Subjects range from rare American Civil War portraits and international travel destinations to black-and-white jazz photos of the 1930s and 40s recently added to the public domain. (A comprehensive report of the first nine months of the project is available online (Springer et al, 2008).

Quantitative metrics

In 2010, Flickr reported that the Library's account had an average of 31,387 views and 31 comments per day, with a total of over 27 million views since launch. Initial interest in the first months of 2008 was very strong; no doubt due in part to the novelty of the Library sharing content in this Web 2.0 forum. Of greater interest is the sustained high level of community activity that followed.

The Library tracks additional statistics, including favorites, comments, tags, group acceptances, and notes. As of October, 2010, for 10,514 items in 14 sets

  • 9,190 items had been selected as favorites 141,327 times
  • over 29,000 comments had been left on 7,800+ images by 10,216 unique Flickr accounts
  • more than 130,000+ tags had been added by 3,507 unique Flickr accounts (9,500 images have at least one community-provided tag)

By October 2010, more than 31,000 Flickr members had chosen to make the Library of Congress a "contact" creating a photostream of Library images on their own accounts. These contacts, supplemented by RSS feed subscriptions, allow the Library to draw attention easily to new uploads and sets throughout the year. One of the Library's goals in placing content on Flickr was to increase the awareness of these photos and these numbers support that conclusion. Quantitative data of this nature does not tell the whole story, but remains a quick reporting method to communicate effectively with senior managers about the project's ongoing value.

Qualitative measures

In addition to building awareness of photo collections (measured primarily through views and favorites), another project goal was to investigate how and if a community might provide new descriptive information through a pool of virtual volunteers. Certainly, many of the comments are personal reactions, such as the typical fan comment "Wow! Cool Pic," and personal reminiscences. However, a stratum of Flickr members is persistently interested in providing historical context to the photos and documenting the sources of that information. We built a set called "Thank You, Great Comments" to make the depth of contribution more visible through examples (http://www.flickr.com/photos/library_of_congress/sets/72157623212811048/).

Another qualitative measure is the degree to which Library staff directly engages with the community. From the outset of the project, Prints and Photographs staff members have responded in approximately 2,000 comments, thanking Flickr community members for their input, drawing their attention to other similar photos in the Library's collection and re-enforcing community interest in contributing by noting that the Library will use their contribution to correct or enhance the Library's records.

Rather than import comments and tags directly into the Library's catalog, we verify the information then use it to expand the records of photos that had little description when acquired by the Library. As of January 28, 2011, the Library has enhanced and updated 2,893 records in the Library's Prints & Photographs Online Catalog (http://www.loc.gov/pictures) citing the Flickr Commons community as the source of the fuller names for people, places, and dates. The symbolic 2,500th record, a photo from the Bain collection of Captain Charles Polack (http://www.loc.gov/pictures/item/ggb2005015681/), is illustrative of the updates taking place based on community input. The new information in the record of this photo now includes his full name, death date, employer, and the occasion for taking the photo, the 100th Atlantic crossing as ocean liner captain. An additional note added to the record points the Library visitor to the Flickr conversation and more of the story with references to gold shipments during WWI.

Qualitative measurements, like level of engagement, are a challenge to gauge and convey. While resources expended are sometimes viewed as a cost, in this case they indicate benefit. The staff time and workflow required to interact with the community ranging from 10 to 15 hours a week at present involves verifying information and updating Library records and ebbs and flows with the level of substantive activity on the account such as the introduction of new collections. Generally, the greater the community engagement, the greater the resource commitment.

Like all Commons members, the other qualitative measure we value highly is the sheer inventiveness of Flickr members who engage with the photographs. One group created an entire community based site called Indicommons (http://www.indicommons.org). Others have provided then-and-now pairings of historic and contemporary sites, combined photos with timelines and maps, and created paintings and other projects inspired by the photos. We also see our photos used to write new articles for Wikipedia, to illustrate subject-based blog posts. This is trackable via Google alerts.

Powerhouse Museum

The Powerhouse Museum was the first museum, and the second institution, to join the Commons project going live 8th April 2008. The first upload consisted of 200 reproductions of high value photographs from the Tyrrell glass plate negative collection that the Museum acquired in 1985. The full collection consists of 7,903 negatives with less than half of the collection currently digitised and mostly "not-fully catalogued" depicting scenes of the city of Sydney and surrounding suburbs. Prior to the Commons, 281 of these images were available on the Powerhouse website. The account currently has 1,811 images with 29 different sets including over 1000 of the Tyrrell Collection and several other photographic collections.

Making the Tyrrell collection richer through various forms of tagging, comments, research, and innovation was a prime motivation for joining the project. It was the first time a collection from the Powerhouse archive was made available for use and re-use under "no known copyright restrictions."

View statistics on the first batch of image uploads exceeded the entire previous year of views of the same images on the Powerhouse website within five weeks. The Powerhouse is now averaging 2,500 views a day over 1,811 images and the total views have nearly reached 3 million. Every image uploaded to date has been tagged at least once, 1,032 images have comments and 1,454 have been "Favorited."

Since July 25, 2008 tags added on Flickr have been re-ingested back to the Powerhouse's own collection on its website to enhance our online search and discovery of our photographic collections. However to date the comments added to images have not been imported in bulk - and have instead gone through a slow curatorial review process and manually processed. A number of minor corrections have been made to collection records, and a few significant discoveries made as a result of comments and engagement on Flickr. Because of the lack of bulk comment importing, the majority of social interaction around these images on Flickr remains outside of the Powerhouse's own systems and is currently seen as rather ephemeral and is not being actively preserved. For most comments this is probably appropriate action for a museum (compared to an archive or library).

A good example of the type of fascinating social interactions that are difficult to consider preserving outside of Flickr is apparent around our image "Portrait of an articulated skeleton." This image currently has 481 comments and is covered almost entirely, pixel-by-pixel, with "notes." This type of interaction demonstrates the flocking behaviour of the Flickr community at large that occurs from time to time; however, these sorts of interactions may not be useful or deemed necessary to be ingested into the collection records.

More broadly the Powerhouse's involvement in the Commons on Flickr has significant organizational effects. Photographs in our collection have been converted from an "archive" status into "collection" status within our catalogue as the museum has searched for more interesting photographs to upload to the Commons. Also as a result of the apparent audience being activated around our photographic collections, exhibitions (Bruno Benini 2010) and photographic competitions (Trainspotting 2010, Trainspotting 2011) have been fast-tracked. This is having continuing impact on how the public perceives the Powerhouse's "brand" - traditionally very rarely associated with photography.

The Powerhouse's content in the Commons on Flickr has also made it easy for collection-based mashups to be made. The earliest of these would be Paul Hagon's Then & Now mashup with Google Street View (August 2008 and later extended to many other Commons institutions). Others include Mob-Labs' work in Layar for the Powerhouse (October 2009), and the Australia Broadcasting Corporation's Sydney Sidetracks (November 2008) which discovered images for use in its project through the Commons. The Flickr API has made many of these possible, and the most recent high value use of the Powerhouse images in the Commons has emerged through a partnership with Sepiatown (October 2010). Sepiatown manually corrected geotags and compass headings on 500 images from the Powerhouse collections in the Commons for ingestion into its Then & Now engine and delivered the corrected data back to the Powerhouse.

Whilst the Powerhouse continues to periodically add new images to the collection on Flickr, the museum is considering how best to manage and sustain the community engagement required and, as Croucher (2010) claims, is "now expected" by a core of enthusiasts on Flickr:

Attempts have been made to use these contributions to improve the Museum's professional collection documentation with varying degrees of success. Though this completes the circle of gift exchange, the principle of reciprocity requires all parties to continue to give and receive in order to maintain the social bonds the gifts have created. In the case of the Powerhouse Museum, the sustainability of their gift exchange relationship with the Flickr community puts the productive capabilities of the Commons in jeopardy.

Smithsonian

The Smithsonian Institution (SI) joined Flickr Commons in June of 2008 with 900 images at the start. To date, there are over two thousand images in the Smithsonian's photostream. The Smithsonian is comprised of 19 museums, 9 research centers, and numerous archives and libraries. The Smithsonian has an umbrella account for all its divisions. Joining the Flickr Commons was a pan-Smithsonian effort which included staff from Archives, Libraries, Museums, and Central IT. Ongoing, the Web and New Media team at the Smithsonian Archives manages participation (approximately 15 hours/month across 3 staff members), coordinating across the Institution with the different divisions, with programming assistance from the Office of the Chief Information Officer.

Currently, the photostream averages approximately 4,480 view counts per day. It is not feasible to directly compare this statistic with the view counts on Smithsonian's websites, as the collections on the Commons are dispersed on several Smithsonian websites. A good example, however, of increased awareness of the Smithsonian's collections comes from the Smithsonian Libraries' "Portraits of Scientists" (http://www.flickr.com/photos/smithsonian/sets/72157605338975676/) set on Flickr. These photographs of 19th and early 20th century scientists and inventors have been available on the Smithsonian Libraries' website since 2003. Though a popular and cited Web resource, in the three months that the photographs had been on Flickr, they received nearly as many visits as during the previous five years on the Smithsonian site. As an indicator of level of interaction, 55% of photos have comments and 89% have been "favorited".

Tracking Flickr statistics involves the following activities:

  1. A spreadsheet tracking monthly uploads, number of views, number of photos with comments, and number of photos 'favorited'. We note significant spikes and potential related activity.
  2. A Google newsfeed to track press related to Flickr Commons with PDFs of significant press mentions.
  3. We are pulling comments and tags for archiving purposes; however, we are not displaying tags on our institution's websites. Any comments we receive with significant information that could inform the image's metadata are forwarded to divisions. Incorporation of this information varies greatly across the divisions.

Since the Smithsonian's launch on Flickr Commons, several significant case studies emerged that highlight engagement with the Commons community.

Using the Flickr Commons to improve collections data and increase online audience

The Smithsonian Archives, as is the case with many archives, seeks to describe its material at the collections-level with finding aids. In some cases, the staff feels a collection has significant value to researchers and the general public to go beyond the collection-level. The Flickr Commons has increased the Archives' ability to describe collection images at the item-level.

In March, 2009, the Archives posted eight unidentified photographs from the Science Service morgue file (SIA Accession 90-105) on the Commons. Tammy Peters, Supervisory Archivist, wrote a blog post for THE BIGGER PICTURE asking for visitors to help identify these women (2009). By July, 2009, Flickr user, "rockcreek," had identified Elizabeth Sabin Goodwin (citing a link to a newspaper clipping).

Figure 1. Flickr user "rockcreek" identifies Elizabeth Sabin Goodwin citing a link to a newspaper clipping.Figure 1. Flickr user "rockcreek" identifies Elizabeth Sabin Goodwin citing a link to a newspaper clipping.

In April 2010, Elizabeth Sabin Goodwin's granddaughter, Linda Goodwin Eisenstadt, contacted the Archives through Flickr, filling us in on more details of her fascinating life (Peters, 2010a).

Figure 2. Goodwin's family leaves a comment about the photo.Figure 2. Goodwin's family leaves a comment about the photo.

Eisenstadt corresponded with the Archives and provided information that allowed Smithsonian Archives fellow, Marcel Chotkowski LaFollette, to further her research on the Science Service collection. As blog commenter Penny Richards, summarized, "To see this story go from an image with initials to a full biography with images and living memories, through crowdsourcing, is wonderful, one of the very coolest parts of the whole Flickr Commons project for me."

Results

  • Enhancing data through crowd-sourcing: Four of the eight unknown women in the Science Service Collection were identified with verifiable sources.
  • Increasing the Archive's audience:
    • Flickr.com is the blog's #13 referring site. These visitors have a very low bounce rate, view more pages on the blog than the average visitor, and spend more time than the average visitor on the site as well.
    • The Flickr Commons has been the subject of many of our blog's most popular posts, and accounts for 23% of SIA's overall press.
  • Increasing use of SIA's resources: The Flickr Commons Women in Science set has successfully increased traffic to and use of the Science Service Finding Aid (RU 7091). Unique page views of the Finding Aid went from the single digits to an average of 78 views a month.

Participation in the Commons leads to significant donation to the Smithsonian Archives

In September, 2008, a collection of previously unpublished photos documenting the Tennessee v. John Thomas Scopes Trial (http://www.flickr.com/photos/smithsonian/sets/72157607580371997/with/2898310107/) was uploaded to the Commons. Fuzzy and in black-and-white, these photos conveyed much information about the experience of and buzz surrounding the trial. Our photo "views" exploded with numerous notes of appreciation. Nearly a year later, Ms. Henrietta Silverman Jenrette, who discovered the photos on the Commons, contacted us through our website requesting we accept her donation of 10 photos of the trial from her father who attended with his previous high school biology teacher. The reason? "I appreciate the way the photos the Smithsonian has are available online for all to see." Peters provides more detail about the donation on the Smithsonian's blog (2010b).

The New York Public Library

The New York Public Library (NYPL) joined The Commons on Flickr on December 17, 2008, starting with 1,300 images from across the library's diverse collections. Images include modern dance photographs, early cyanotypes - the first photographic works by a woman - by Anna Atkins, geographic surveys taken for the Union Pacific Railroad, Berenice Abbott's 1930s Federal Art Project documenting New York, 19th century photographs of Egypt and Syria, and more. Today there are 2,525 images from the library's research centers, including the Schomburg Center for Research in Black Culture; the Library for the Performing Arts; Science, Industry & Business Library; and various research collections from the Stephen A. Schwarzman Building.

The library's digital staff, in collaboration with staff from specific collections, chooses thematic sets of images from NYPL's digital collections (http://digitalgallery.nypl.org). An API-enabled CMS pulls metadata for collection items and formats the records for uploading to a staging account. Tags - based on metadata from subject headings - are automatically generated during this process. Once images have been cleared and curatorial descriptions written for each set, the images and their associated metadata are uploaded to NYPL's public Flickr account.

Measuring quantitative usage

In the first 18 months NYPL's Flickr account recieved 2.1 million total views, over 11,000 "favorites" (55 percent of all images), 4,664 contacts, and 3,695 user-contributed tags. More recently average traffic includes around 2,000 visitors and 4,000 page-views daily. The library's digital collections site by comparison recently averaged 6,000 daily visitors and 64,000 page-views. Most library home pages for collections represented in The Commons average under 50 daily views.

Traffic statistics, especially since the release of the Flickr statics API, have been robust enough to measure most activity around photos within Flickr's ecosystem. The library has begun tracking NYPL traffic on other 3rd-party platforms with tools such as HootSuite, which records viral activity around tweets. But gaining access to metrics like HootSuite or Google Analytics' campaign tracking integration is more difficult for Commons items since the primary focus of Flickr statistical reporting remains on activity within Flickr's platform or APIs.

Referrals from Flickr to NYPL's digital collections have consistently been in the top 10 for external sites. While the incoming Flickr traffic is not high, the 38% bounce-rate - users who exit after a single click - shows that Flickr visitors who link back to NYPL are particularly engaged when introduced to the library's digital collections. Time spent on-site is over 6 minutes, the highest for top 10 referring sites.

The library's Strategic Planning Office has created a social media "dashboard" which allows some comparison of statistics across platforms where NYPL content has been uploaded. Flickr contacts, in this context, can be compared to subscribers on other platforms such as Facebook. The social media dashboard allows administrators to gain a high-level view of usage patterns across multiple social media sites, including Flickr statistics where they are available. (see figures 3 and 4)

Figure 3. View of subscribers across several platformsFigure 3. View of subscribers across several platforms

Figure 4. iTunes, YouTube, and Flickr downloadsFigure 4. iTunes, YouTube, and Flickr downloads

Measuring qualitative usage

One the surprising patterns we have observed is what users have been able to tease out of content that all ready exists on The Commons. Some have done this through closer examination of nearby or related images, or by relying on personal knowledge (e.g. "that is my great-uncle in that picture"). Examples include:

  • A user asked anyone knew the cause of an injury shown in a U.S. Civil War medical photo; Another user answered with a specific diagnosis found in another link to a Commons image of the same soldier. (http://www.flickr.com/photos/nypl/3110015223/)
  • A user provided a recollection that proved a prominent Yosemite Valley location was being portrayed upside down, due to a lakeside reflection in a 19th Century photo. The library later discovered the JPG derivative had been incorrectly "flipped" during processing. (http://www.flickr.com/photos/nypl/3109103219/)

Other users have "gone the extra mile" in interacting with images. A user, for example, has posted local narratives, explanations, and history about 19th century Japanese photographs in NYPL's collections, including before/after images taken at locations pictured in old photographs (http://www.flickr.com/photos/nypl/3110701692/). With limited exceptions - misspelling, photo aligned in wrong direction - the library has not yet begun integrating content back into item records, but these additions remain important as a record of substantial and engaged user activity around digital collections on The Commons.

It can be difficult to measure impact the further from the Flickr Commons platform that activity around an image occurs. For example 3rd-party applications that display data from across a merged collection of Flickr images, such as the Commons Explorer site (http://creative.canberra.edu/au/cex/), may not offer reporting mechanisms outside of the hit-counts on specific images that are tabulated by Flickr.

Flickr Commons and beyond

Uploading items through Flickr's API and monitoring overall Commons activity currently requires on average only a few hours per week to sustain. But responding to platforms like Flickr on a long-term basis is a broader policy challenge institution-wide. When simple responses are needed our digital team has reached out to curators and staff, but ideally dedicated staff roles could be allocated to Commons content and metrics. With the creation of a social media task force and new activity across the library on multiple platforms, NYPL has been engaging more directly with external sites like Twitter and Facebook. These activities are enabling NYPL to leverage its experience, creating new policies and expanding staff roles to respond not only to The Commons, but also to other social media contexts in a more agile way.

Cornell University Library

Cornell University Library started contributing images to Flickr in June 2009, in a pilot project started by the "Library Outside the Library" committee. The A.D. White Architectural Photograph Collection was the first collection uploaded in 2009 with approximately 1,500 images. Even before Cornell officially joined the Commons, hit counts on the A.D. White images on Flickr were significant enough to decide to continue investing in the project.

Cornell University Library officially joined Flickr Commons on May 19, 2010. Today, its photostream includes 3,972 images. Cornell's main goal of using Flickr was to increase the visibility of the collections, not primarily to solicit crowdsourced descriptive information about the images. Daily hit counts now average around 1,000 per day. Users have commented on 349 of the images, and 1,130 have been added as "Favorites", 231 images have been added to 141 user galleries, and 542 users have added CUL as a "Contact". The project has grown since its inception in 2009; now three groups in the library have some responsibility for maintaining and managing Cornell's Flickr presence.

Evaluating and understanding the impact of Cornell's Flickr images surfaced multiple issues: it was a challenge to gather usage data from Flickr that matched what was available for locally hosted collections yet accommodated Flickr's "social statistics", tracking use of the collection beyond Flickr was not straightforward, and it emerged that monitoring interaction with the images was time-consuming and often tricky to report back to the institution.

Handling quantitative statistics

Flickr is a third party platform; Cornell no longer has complete control over the format and delivery of the statistics for their images hosted there. The statistics available from Flickr were different than the statistics Cornell was accustomed to for its digital collections because Flickr is structured to support social interactions between users, including crowdsourced metadata for individual images, unlike Cornell's locally hosted digital collections. Even though Cornell was aware that hit counts didn't reveal the complete story of how users were interacting with our Flickr images, they were still important statistics to collect. When Flickr released an API to gather statistics programmatically, there was a significant time commitment to develop the code to format the data from Flickr so that it was compatible with Cornell's existing Web logs infrastructure. Even then, there were gaps in what could be learned about Flickr users from these statistics. Cornell's internal Web statistics system, for instance, allows library staff to distinguish between "on campus" and "off campus" hits and determine how long a visitor has been on a particular page. Flickr does not provide IP addresses of visitors or session information, so this kind of information is not available to Cornell library staff from the Flickr collections.

Handling qualitative measures

There were certainly gaps in Flickr's statistics compared to Cornell's own server logs and there was also usage data from Flickr that could not be accommodated by the "server log" framework. Many visitors to Cornell's Flickr photostream add tags, comments, and notes to images or mark them as "favorites" and add Cornell University Library as a contact. Library staff pointed to these activities as far better indicators that users had significantly engaged with the images than just visit counts. While it is straightforward to collect this information using the Flickr API, there is currently no way to store these "social" statistics within Cornell's existing Web statistics system. Institutionally, the challenge is to incorporate this information into Cornell's reporting systems. Ultimately, placing images on Flickr meant giving up some level of control over the way statistics are collected and ultimately reported.

How to measure "significant engagement" surfaces again when considering how to evaluate the use of the Commons images beyond Flickr. It was necessary to develop a multipronged strategy for locating instances where users had referenced our images. Currently, Cornell's tracks this information through Flickr's referral URLs, a Blogpulse feed (http://www.blogpulse.com), and a Google alert (http://www.google.com/alerts). Still, there are limitations to this approach as it does not capture everything. If a user does not link back to Cornell's Flickr record, it is nearly impossible to discover that they are using one of Cornell's images. Additionally, there is a distinction between pages that include images because the site administrator is simply displaying Flickr images tagged a certain way using the Flickr API (e.g., http://www.wordnik.com) and a blog author who is spotlighting a specific image from Cornell's collection because it aligns with a theme in their writing. Unfortunately, it is very hard to distinguish these situations without human intervention. Resourcing this is a significant challenge even though it provides some of the richest examples of interaction. Additionally, how should Cornell save examples of this level of interaction when it does locate them? Currently, Cornell saves links to examples of reuse on delicious (http://www.delicious.com) and displays them as a feed on its website. While this system is not perfect, it does allow for some way to track this information.

Ultimately, measuring the impact of the Flickr project will continue to evolve as users find new ways of engaging with the collections. Cornell will also need to continue to explore new ways of formally documenting the use of its Flickr Commons images.

Conclusion

The level of engagement and interest in the collections on Flickr Commons certainly justifies the effort required to make these image collections available on this platform. For some institutions, information provided by the community has proved invaluable, as comments and notes have been incorporated back into the institution's own records. Developing metrics that reflect this new landscape is certainly a challenge, especially understanding how users are interacting with the collections and provisioning the necessary resources to engage with that community. It is imperative to continue this conversation as social networking platforms emerge and evolve.

References

Crocher, A. (2010). Cultural heritage photographic collections and a circle of gift exchange: The Powerhouse Museum joins The Commons on Flickr. University of Wollongong: Honours Thesis.

Peters, T. (2009). Who are you? Blog post in "The Bigger Picture". Consulted January 31, 2010. Available http://blog.photography.si.edu/2009/03/30/who-are-you/

Peters, T. (2010a). Finding Elizabeth. Blog post in "The Bigger Picture". Consulted January 31, 2010. Available http://blog.photography.si.edu/2010/07/07/finding-elizabeth/

Peters, T. (2010b). New Donation of Scopes Trial Photos to the Smithsonian Archives. Blog post in "The Bigger Picture". Consulted January 31, 2010. Available http://blog.photography.si.edu/2010/02/03/new-donation-of-scopes-trial-photos/

Springer, M., B. Dulabahn, P. Michel, B. Natanson, D. Reser, D. Woodward, and H. Zinkham, (2008). For the Common Good: The Library of Congress Flickr Pilot Project. Consulted January 31, 2010. Available http://www.loc.gov/rr/print/flickr_report_final.pdf

Cite as:

Bray, P. et al., Rethinking Evaluation Metrics in Light of Flickr Commons. In J. Trant and D. Bearman (eds). Museums and the Web 2011: Proceedings. Toronto: Archives & Museum Informatics. Published March 31, 2011. Consulted http://conference.archimuse.com/mw2011/papers/rethinking_evaluation_metrics