New Techniques for Visualizing Large Image and Video Collections

Many museums and media archives have already digitized large parts of their collections, and we may expect even more images and video to become digitally available in the near future. Are the standard web and desktop interfaces for accessing digital collections adequate for understanding these collections? Popular interfaces for showing images and video such as list, gallery, grid, and slide do now allow us to see the contents of a whole collection. These interfaces usually only display a few items at a time, regardless of whether you are in a browsing mode, or in a search mode. Because we are not able to see a collection as a whole, we can’t compare sets images or videos to each other, notice patterns of change over time, or understand parts of the collection in relation to the whole.

Graphing and visualization tools offer a range of visual techniques designed to reveal patterns in data. However, these tools have their own limitations. A key principle, which underlies the creation of graphs and information visualizations, is the representation of data using points, bars, lines, and similar graphical primitives. This principle has remained unchanged from the earliest statistical graphics of the early 19th century to contemporary interactive visualization software which can work with large data sets. Although such representations make clear the relationships in a data set, they also hide the objects behind the data from the user.

Since 2008, our lab (Software Studies Initiative, http://www.softwarestudies.com) at University of California, San Diego (UCSD) has been developing visual techniques that combine the strengths of media viewing applications and graphing and visualization applications. Like the latter, they create graphs to show relationships and patterns in a data set. However, while plot making software can only display data as points, lines or other graphic primitives, our software can show all the images in a collection superimposed on a graph. We call this method “media visualization.”

To date, we have already successfully applied our techniques to artworks, photos films, video games, comics, magazines, books, and other media content. Examples include all paintings by van Gogh, one million pages from manga series, hundreds of thousands of pages from 19th century American newspapers, and hundreds of hours of video gameplay. Our past and present collaborators include Getty Museum, Library of Congress, Austrian Film Museum, Magnum Photos, Netherlands Institute for Sound and Image and other institutions who are interested in using our methodology with their media collections. Starting in the Fall 2011, we are releasing our fully documented tools we developed under open source license.

In our demonstration, we will show the examples of our visualization and explain our key techniques for creating media visualizations.