Background
Digital heritage and semantic Web technologies hold out the promise of nearly unlimited access to cultural knowledge. The problem is that cultural meaning does not reside in individual objects, but in the patterns of knowledge and events, belief and thought that link them to each other and to the observer. This is why narrative is so important to the communication of, and meaningful understanding, of culture.
Concept and project objectives
DECIPHER will research and develop a reasoning engine, virtual environment and interfaces that can present digital heritage objects as part of a coherent narrative which is directly related to individual information searches and user contexts. This will allow the user interactively to assemble, visualise and explore, not just collections of objects, but the knowledge structures that connect and give them meaning.
Although there is a vast and growing store of digital objects, most of the metadata associated with them is in the form of flat MARC and Dublin Core style records (time of creation, where held), which do not in themselves provide explicit interpretation. We are typically left with a choice of Google-type searches or browsing structures, neither of which convey narrative, or highly-structured, curated Web presentations of fixed bodies of work. Although there are now systems that can show objects in relation to maps and timelines, and we can 'fly through' geo-tagged images on Google Earth, these do not explicitly represent, nor necessarily convey, the conceptual patterns across space and time and their possible explanations. While many cultural websites (such as the 'Essential Vermeer') are extensive and elegant, their narratives are 'burnt-in' and fixed by their authors: they can never be complete, and there is a good chance that the searcher's interests will not be met. They do not recognise the user's focus of interest, they do not provide directions to individual explorations, and they do not pull in additional material from elsewhere at need. Finally, current systems are tied to the technology platforms for which they are authored. Web-based approaches offer a two dimensional sensory experience that is limited in comparison with the real world; immersive and haptic systems still require expensive special-purpose hardware whose use is restricted to small numbers of people at the heritage sites themselves.
DECIPHER proposes new solutions to the whole range of narrative construction, knowledge visualisation and display problems, with a high degree of future proofing. We already know, from work carried out by KMi in the SILVER project, that we can uncover interesting patterns from data by using a combination of machine reasoning, knowledge visualisation and user tagging. Even flat metadata associated with objects can support reasoning techniques to identify and present groupings or classification of objects (for example, changes in women's roles from the late 19th to early 20th Century). However, it currently requires a teacher or mentor to make clear the reasons. DECIPHER will produce a step change in the process by combining much richer, event-based metadata with causal and dynamic reasoning models. By associating an object with a set of events, it will become possible to apply machine reasoning in a much more interesting way, not only to identify clusters of events or objects but also to construct narratives.
There are several ways of generating event-based metadata. It can be professionally created from flat metadata, but this is expensive. On the Web, user-tagging of objects, particularly photos and video, is increasingly encouraged as a means of generating metadata (although semantic inconsistency can be a problem).
By conducting searches within and across collections of objects, extracting and associating event-based semantic metadata, we can feed a knowledge engine with sufficiently rich material to support reasoning over a selection and find clusters of content in time, space and theme.
Clearly, it is not enough to develop a narrative: we must present it in a way that allows the visitor to explore the knowledge space and objects in the richest way possible, taking full advantage of whatever technology platform is available (fixed or mobile, 2D or 3D, window or immersive). The visual formalisms, interface and interaction techniques will draw on DIT's previous work in mobile interaction and virtual museum spaces (ICING, i3g). We can think of the information space as a three-dimensional virtual world, with a range of different user interfaces and templates that tailor the view to the individual's platform and narrative interest. This approach will make it possible to render and explore not only the cultural objects but also their narrative and conceptual relations in a three-dimensional spatial representation. Customised rich interfaces may support digital curation in many ways. For example, some installation art consists of a digital file and the curatorial instructions for "how to show it." A system that links media resources, curatorial metadata and multi-modal interfaces would allow a much deeper exploration of such works. To take another example, three-dimensional audio clues might provide spatial information to people with visual impairment.
What information can be visualised is also a function of the platform: a high- resolution video wall can display different types of information and more complex relationships than a small mobile screen. In the lifetime of DECIPHER, we can expect the introduction of smart phones and pads with HDTV-resolution video, stereo displays, and significant advances in large-scale and immersive display environments. In order to deal with the gamut of current and predicted display technologies, we will develop model-based interfaces that use strict HTML 5 to tailor the presentation and rendering (whether 2D or 3D) to the capacity of the platform. The project addresses the Call Objective outcome d) Adaptive cultural experiences with the aim of creating personalised cultural narratives, knowledge structures, and viewing environments, all based on representations of digital heritage gathered from different sources.
The target outcome will be achieved via six assessable Scientific and Technical Objectives:
- To define and represent curatorial processes that form the basis for cultural narratives by studying museum workflows and curatorial practices for managing and interpreting digital artefacts; providing definitions of semantic processes; building a generic semantic workflow model; preparing a specification for the demonstrator applications to select trial datasets and set up testbeds.
- To research and develop new methods of reasoning about curatorial practice by formalising the description of the curatorial process using RDFS/OWL, CIDOC CRM and higher-level ontologies for time and space; developing machine reasoning methods for defining perspectives, defining transitions and generating narrative structures; providing ways of biasing and filtering the process for different contexts; researching methods of reasoning over successive sets of content to define recommendations and build trajectories across the content.
- To improve methods of identifying and retrieving relevant data and populating the knowledge base from multiple collections and sources by focused crawling and language-dependent pre-processing; content classification and document grouping using semantic analysis and machine-learning methods; semantic annotation and representation of pre-processed documents to extract relevant entries, using templates conforming to CIDOC CRM.
- To design and build a robust system for data gathering, reasoning and narrative generation by defining a system architecture that can take advantage of distributed data storage and processing methods appropriate to networked heritage collections, including cloud computing, SOA and SaaS; designing a metadata database structure; to integrate and code the functions for reasoning, data retrieval and knowledge base population; developing middleware for knowledge structure and content presentation; and building robust prototypes for testing and evaluation.
- To develop standards-based, context-aware narrative interfaces and visualisation tools by creating a high-level interface abstraction that supports the adaptation of interfaces to the client system, use environment and cognitive characteristics of the user; developing software tools and models for 3D interfaces, and temporal media rendering; developing time- and event-driven interfaces that can express the relations between objects, causes and events; and developing interfaces for mobile and location-aware devices.
- To demonstrate and evaluate the technologies in experimental use, across different types of cultural objects, collections and platforms to be achieved by on-site, remote and cross-collection tests with representative samples of users; evaluation in field trials and showcases at partner sites, with realistic data sets and in two languages (English and Italian).
Outcomes and Innovations
- New ways for Curators, Educators and other Cultural Heritage Professionals to mediate their collections and explain cultural objects in context.
- New ways for Users to explore, visualise, reason with, and add perspective to sets of cultural objects, and to share these in visually attractive narrative interfaces, based on standards, supporting many modalities and devices.
- Schema and tools (based on higher-level standards) for describing cultural content resources in terms of their relationships to events, places, people and themes, and the user's purpose and cognitive profile.
- Search, retrieval and aggregation methods and tools that allow cultural content harvested from public sources, across cultural domains (art, music, dance etc.), to be semi-automatically described with the event schema.
- Reasoning methods and software tools that support an individual user's organisation and exploration of content using recommendation and trajectories (using contrasting metadata) across historical periods and cultural domains.
- A software service of Open Source components tested in real world conditions, which integrates these innovations into a single, robust, and sustainable repository of narrative that is linked and adds value to the collections of digital cultural objects held by partner institutions and external networks such as Europeana.
- A suite of standard-based user interfaces that deliver a personalised, rich media experience that takes full advantage of any browser-based devices' capabilities, including 3D and stereo, to deliver the richest possible experience.
Fig 1: DECIPHER results overview
Progress beyond the state-of-the-art
Digital Curation
The way in which museums are structured and administered has become increasingly complex in recent decades, reflecting changing ideas about the relationships between art and culture, exhibition spaces and audiences, and responding to developments in digital media. Museums now operate within the domain of knowledge management; they collect objects, document and frame views of the primary collection and the wider world. Making the resources of a museum accessible to individuals and communities requires a broad range of expertise encompassing collection management, research strategies and methodologies, databases, legal and ethical issues, digital preservation, and cultural heritage information networks. The curator has become an intermediary, working across contexts and disciplines within the gallery and the public realm, mass media and online technologies. Curatorial practice has "become an increasingly diverse, collaborative and creative activity - the dynamic interface between... art and its audiences".
There is a growing tendency to see the museum less as a temple of culture than as a public forum. The idea of the museum as a laboratory or public forum extends naturally to the exploitation of on-line activity in virtual spaces. Museums such as IMMA have expanded the concept of the museum space to encompass first, its galleries and surrounding grounds, second, its onsite studios for resident artists, and third, the regular touring of its collection through national and international programmes; now, the Web becomes a fourth space.
There is a growing interest in the impact of curatorial process in the construction of meaning: Intervention Art highlights the assumptions made within the museum narrative. In Mining the Museum (Maryland, 1992) Fred Wilson brought objects concerned with slavery out of the reserve collection and juxtaposed them with the permanent display, to present an alternative historical perspective, seemingly repressed by the museum narrative. Eduardo Paolozzi's Lost Magic Kingdoms at the British Museum questioned the museum's selection process by showing previously hidden ethnographic objects that could be thought of as unworthy of public display. IMMA's exhibition What happens next is a secret asked what happens when artworks become part of a collection and are subsequently shown in different contexts? The artworks were changed regularly during the course of the exhibition, the removal of pieces generating absences which call to mind gaps in our memory and point to the partially hidden nature of museum collections.
Curatorial thinking is generally turning in directions that look beyond the boundaries of the museum and its collections, in ways that imply the use of information from the wider Web to create revelatory access: "...it is interesting to think about the museum as an archipelago. The idea of non-linear time implicit in this idea or this concept, the co-existence of several time zones, would of course allow for a great variety of different contact zones as well".
Progress beyond the state of the art
Although there is a growing body of research into museum practice in the Digital Humanities, and in Digital Library and Information Technologies fields, the majority of work is descriptive rather than quantitative. The construction of digital exhibition spaces and personalised access to museum content has, so far, been somewhat of an ad hoc undertaking. DECIPHER will advance the state of the art by analysing curatorial and educational museum practices to build a general, semantic model of museum practice and curatorial workflows. This will create a formalism that is susceptible to machine reasoning to be incorporated in the DECIPHER 'story-ware'.
Consortium
DECIPHER is a three-year, €4.3 million Specific Targeted Research Project (STREP), aiming to develop new solutions to the whole range of narrative construction, knowledge visualization and display problems for museums.
Project details
- Project coordinator: Eoin Kilfeather, DIT Dublin, Ireland
- email: ekilfeather@DMC.DIT.IE
- START DATE: January 2011
- END DATE: December 2013
- FUNDING SOURCE: European Commission co-funded project
- http://www.cordis.europa.eu/ist/kct/fp7.htm
Project partners
- Dublin Institute of Technology
- National Gallery of Ireland
- Irish Museum of Modern Art
- Open University,
- System Simulation Limited
- Brno University of Technology
- Alinari 24 ORE SpA
Project co-funded by the European Commission within the Seventh Framework Programme: January 2011-December 2013
