Virginia Gow, Lewis Brown, Courtney Johnston, Andy Neale, Gordon Paynter, and Fiona Rigby, Digital New Zealand, National Library of New Zealand, New Zealand
Digital New Zealand aims to help make New Zealand content easy to find, share and use. The project team has chosen an approach which involves identifying and attempting to solve the challenges faced when creating, describing, licensing, preserving and making digital content available: every part the project team’s work is targeted at creating a solution to a challenge. The first wave of the project produced three innovative tools: Memory Maker, for making short videos using researched New Zealand material; a customisable search builder, for gathering data; and an open API for connecting DigitalNZ metadata with other data sources. In this way, DigitalNZ is showing what is possible when you improve the accessibility, usability and discoverability of New Zealand content. There will be more to come.
Keywords: cultural heritage, collaboration, Web 2.0, API, metadata, open content, DigitalNZ, Web services, remix
If you ask the average Northern Hemisphere citizen to locate New Zealand on a map, you may begin to get a sense of why many New Zealand cultural organisations find it difficult to make their content visible in the vastness of the Web.
New Zealand is a small island nation with a population of just over four million people (around 0.06% of the world’s population). It is noted for its geographic isolation, natural beauty, and ‘can-do’ attitude.
In this paper, we describe some practical solutions the Digital New Zealand programme has developed to confront issues that make New Zealand content difficult to discover, share and use; and discuss how we are working with a range of cultural heritage institutions in New Zealand to provide access to collections.
DigitalNZ is testing ways to create digital content, collect and share existing digital content, and build smart, freely available search and discovery tools. The project is working with New Zealand organisations, communities and individuals to aggregate their metadata and help make hard-to-find content available for discovery, use and re-use in unique ways.
- The Memory Maker lets people craft their own commemoration of the 90th Anniversary of the Armistice by remixing film, photographs, objects and audio clips into a short video that can then be saved, shared, and embedded.
- A customisable search builder lets people design their own search widget to find the type of New Zealand content they are interested in and embed it in their blogs and Web pages.
- Finally, an open API enables developers to connect the metadata that fuels DigitalNZ with other data sources, enabling new digital experiences, knowledge, and understanding.
We draw particularly on the programme’s first activities, which used the theme of “Coming Home” to focus content development, based on the commemoration of the 90th Anniversary of the World War One Armistice and the return of soldiers to New Zealand.
Digital New Zealand, however, has a much wider reach across collection, content and topic boundaries – including into the geospatial, educational, and research sectors, and indigenous cultures.
Early last century, when the population of New Zealand was just over one million, many New Zealanders’ first experience of leaving the country was to fight in the First World War. From that 1914 population of just over one million, the War took more than 100,000 New Zealanders overseas. Far fewer came home, and nearly half of those who did were wounded.
As with many other countries, the impact of the First World War on New Zealand lasted well beyond the Armistice in November 1918 – a fact recorded and documented in many of the country’s cultural and heritage institutions, not to mention in the fabric of the community and through the presence of memorials in almost every town.
Yet much of the wealth of New Zealand content that carries these memories – often already made digital and stored in collection databases – is hidden or buried, especially in an indexed international Web of one trillion pages (Singer, 2009) which simply scrapes the surface and dilutes content from small nations or specialised niches.
If content is hidden or buried, it can’t be used to make new ways of remembering. As cultural thinker Charles Landry (2007) puts it, “heritage works best when we perceive ourselves to be part of its continual creation”.
Memory Maker: Remixing cultural heritage content
Memory Maker blurs the line between consuming and producing content. What’s sometimes called
‘remix culture’ […]. Digital technologies have opened up new possibilities for young
people to access and represent the stories of their culture by taking sound and
images and recombining them to say something new, something relevant to them.
(Sarah Jones, Lunch Box: Software & digital media for learning, November 2008) http://lunchbox.org.nz/2008/11/get-coming-home-on-your-schools-website-wiki-or-blog/)
The interactive Memory Maker provides a real-world example of how licensed or out-of-copyright New Zealand content can be remixed by users to create new perspectives and experiences if content is appropriately identified.
Inspired by previous Museums and the Web conferences, we wanted to let New Zealanders create their own way of remembering the impact of war by finding and creatively reusing (without restriction) digital content from organisations, individuals, and communities.
The Memory Maker is an interactive on-line video remix tool that lets people mix historical film footage, digitised photographs and objects, and music/audio clips into a 60 second video that can be saved, shared, and embedded on other sites.
How it works
The Memory Maker uses the Ideum Editor One software application. The problem we were trying to solve here was not software development, but lack of understanding and adoption of new licensing and rights practices by cultural heritage institutions.
Using an off-the-shelf product allowed us to focus on working with members of New Zealand’s National Digital Forum - a coalition of museums, archives, art galleries, libraries and government departments - to identify material that could be ‘freed up’ for people to remix with permission.
We credit the content providers in our Web space: http://kete.digitalnz.org/site/topics/show/18-memory-maker-coming-home
I made a movie last night with mum. Mum said it was really cool, we should send it to people. That’s what I did (School student, Room 5, Nelson Central School)
Thanks for directing me to room 5's masterpiece. It's really exciting to see our content being used like this and I can't wait to see what else evolves. (Project manager photographic collections, The Nelson Provincial Museum)
The Memory Maker provides a taste of what is possible when collecting institutions modernise their practices for keeping and managing copyright information, using Creative Commons licenses or ‘no known copyright’ statements.
For us, the success of the endeavour was seeing a class of school children use the tool to make movies that put our attempts to shame. Learning about ‘hyperlinks’ today, these young New Zealanders will be the developers and creators of tomorrow.
But it wasn’t always easy to convince institutions to try something new. Later in this paper we describe how we worked with content providers to take up this opportunity.
Exploring the Coming Home Collection
The “Coming Home” theme provided a focus for New Zealand’s National Digital Forum to work together to enhance electronic access to our culture and heritage.
The Memory Maker uses only a fraction of the Coming Home content. The full set of contributions is accessible through a Coming Home search tool, occasionally on a google-like hosted search page (Figure 5), but more often through a search widget embedded on many New Zealand Web sites (Figure 6).
The result is unique New Zealand content being brought together and accessed through discovery points all over the Web, from educational blogs through to institutional Web sites, wikis and intranets – and if people can’t add the search widget to their site, they can link directly to a hosted version of the search.
What the user experiences
The following screenshots give a taste of what users experience when they come across the Coming Home search tool. Figure 7 shows the Coming Home search box (‘powered by DigitalNZ’) embedded on a community repository Web site that has also provided content for Coming Home.
Figure 8 shows the user searching Coming Home for ‘flu epidemic’. One of the tragedies of the Great War was that in the two months following the declaration of peace, New Zealand lost about half as many people to influenza as it had in the whole of the First World War – and one of the ways the disease spread was at memorial events.
The first results are embedded in the Web page, and the user can click on a result to go straight to the item on the content provider’s Web site – or choose to see more results.
In Figure 9, we see what happens when the users select ‘Show more results’. They are taken to the ‘hosted search result’ page on the DigitalNZ Web site. This provides a full interface to the Coming Home collection. Note the faceted filtering allowing further search refinement and manipulation of information, including the License facet.
Figure 10 shows a search result selected by a user. In this case the content was created by a small provincial museum, and is hosted by the DigitalNZ shared repository, Kete Digital New Zealand.
Kete (the Māori word for “basket”) is an open source community repository developed by New Zealand company Katipo. DigitalNZ hosts a Kete to support institutions that lack the resources to otherwise get their content on-line.
The final step in our user’s progress is to follow a link from Kete Digital New Zealand to the Nelson Provincial Museum Web site (Figure 11).
DigitalNZ is driving users from all over the Web to content providers’ Web sites, where they can get more detail about the content and the institution that provides it.
Widgets for everyone
The Coming Home collection was a timely example of what the DigitalNZ technology can do, but it is only the start of the DigitalNZ programme. In fact, the Coming Home content currently represents only 10% of the material we have brought together. There is huge potential if we let other users build search widgets for their Web sites as we have for Coming Home.
Make your own Search Engine
Wow! You can see the educational uses immediately. Pop a search on your wiki or Web site relating to your class’s inquiry or project - learners will be guaranteed quality New Zealand content on that topic. (Sarah Jones, Lunch Box: Software & digital media for learning, December 2008) (http://lunchbox.org.nz/2008/12/liberating-our-digital-content/)
The customisable search builder application lets users build a search engine across information in DigitalNZ to find the New Zealand content they are interested in – in 3 easy steps:
On saving the search, they can also:
- Link to a hosted version of the search
- See the search in the Widget gallery (http://digitalnz.org/widget-gallery)
- Share the search in the context of their own content such as a blog or Web site
- Subscribe to an RSS feed alerting them to new results in the search.
The ability for people to take their designed search tool back into their own space is a crucial feature of what DigitalNZ has developed. We don’t expect the DigitalNZ Web site or search box to be the centre of everyone’s universe - in fact, we initially set out to build an on-line presence with no homepage.
The customisable search builder tool is simple to use, but that doesn’t mean you can’t do some very powerful things with it, as we demonstrated with the cross-collection Coming Home search, which uses advanced Boolean logic.
The examples we have drawn on so far show part of what DigitalNZ is doing, but they don’t explain why or how.
Digital New Zealand ā-Tihi o Aotearoa (also known as DigitalNZ) is a national programme of work that aims to make New Zealand digital content of all types easier to find, share and use.
Origins of Digital New Zealand
Like many nations, New Zealand has developed government strategies to ensure it stays visible and relevant in a digital world, including a targeted Digital Content Strategy ‘Creating a Digital New Zealand’ (http://digitalcontent.govt.nz).
The Digital Content Strategy, led by the National Library of New Zealand, seeks to ‘chart a course for a content-rich digital New Zealand, where New Zealanders are actively engaged in creating, discovering, sharing and using content in a digital form’ (National Library of New Zealand, 2006). In summary:
Creating a Digital New Zealand is about making New Zealand visible and relevant in a connected digital world. It aims to ensure that we are innovative, informed and capable in telling our stories, experiencing our heritage and cultures, and creating our digital future. To that end, an important starting point for the strategy is recognising that the value of content is in what it delivers and enables for end-users (http://digitalcontent.govt.nz).
The Digital New Zealand programme is a flagship action of the Content Strategy. DigitalNZ received ongoing government funding in 2007, and the first project work kicked off in 2008.
But what exactly is Digital New Zealand?
In a nutshell, Digital New Zealand is developing and testing solutions that showcase what’s possible when we really focus on improving access to and discovery of New Zealand content. Figure 13 shows what Digital New Zealand currently does.
Our goal is to uncover hidden or buried New Zealand content such as images, audio, video, interactives, and documents, and make these available for discovery, use and re-use through our services.
The example tools and applications we have discussed so far are ‘powered by Digital New Zealand’, and fuelled by the metadata and content from the many content providers we collaborate with.
But our tools and solutions aren’t just software developments. We are also establishing good practice guidance on creating, describing and managing content, and an on-line service for New Zealand organisations that may have questions about digital content projects or contributing to DigitalNZ.
Collaborators and partners
DigitalNZ is led by a small team based in the National Library of New Zealand. We work with New Zealand organisations, communities and individuals to gather and aggregate information about cultural heritage content in a machine-readable, standards-compliant way; so that new tools can easily be developed to bring the content to life.
We’ve worked with three companies so far (Boost New Media for design, 3Months for front-end development and Codec for back-end development), employing an agile development approach using Scrum. Ideum provided the remix editor for the Memory Maker, and the Aotearoa People’s Network and Katipo worked with us to build the DigitalNZ Kete.
We have been joined by a wonderful variety of New Zealand content providers around the country, including universities, small and large museums, archives, libraries, community initiatives, broadcasters, and government agencies. A list of content providers we have worked with to date is provided as Appendix 1.
Behind the scenes
Our content partners share metadata about New Zealand content with us (including direct URLs to content objects such as photographs and documents), so that we can share that information with everyone through a Web interface, and open it up to other developers - as yet unknown - through an Application Programming Interface (API).
Technically, the Digital New Zealand system is in three parts: a backend, a metadata store, and a front end. The backend harvester is written in Java and some legacy PHP (based on the open-source PKP Harvester project). The metadata store is in MySQL, using Solr to power the search. The front end, including the API, the Web site, widgets and so on, are in Ruby on Rails.
We worked fast and furiously to launch the tools and applications we have described in this paper – and relied on an agile development methodology to complete our first wave of activity within a tight timeframe of 16 weeks: in time to meet the immovable date of the 90th anniversary of Armistice Day, 11 November 2008.
Bringing metadata into a centralised repository
Our approach to gathering the metadata involved:
- Direct harvesting from individual content providers via OAI-PMH, XML sitemaps and screen scraping, or RSS feeds.
- Collaboration with aggregating content and metadata services that content providers already participate in such as Matapihi, KRIS, NZ Museums, , Kete Horowhenua, and NZ On Screen.
- Providing the Kete Digital New Zealand Web site for organisations to use if they couldn’t find an immediate hosting alternative that could easily ‘port in’ to DigitalNZ.
More specifically, we worked with content providers to:
- Harvest, copy, host and/or store their metadata and thumbnail images to allow free access to it via the Internet. This means that more people will know that their digital collections exist, and see links to their items alongside other related content (including through DigitalNZ Search).
- Establish and maintain RSS or similar data feed services for the metadata we collect for their content, so that it is machine-readable and can be discovered by more users through other Web sites and on-line services.
We discuss common questions from content providers in following sections.
Encouraging new tool development through an open API
The coolest thing to be done with your data will be thought of by someone else (Rufus Pollock, Director of the Open Knowledge Foundation, May 2007)
The best plain English explanation of an API (Application Programming Interface) we have heard recently is from George Oates, speaking at New Zealand’s National Digital Forum conference: “an API is basically a way to give developers permission to hack into your database”. It’s a way for us to enable people to make new, exciting tools based on the content we have aggregated.
While open APIs have been around for a while – particularly for large corporations such as Google, Amazon, and Yahoo – there aren’t nearly as many examples of open APIs serving data from cultural heritage organisations, especially over 25 unique collections brought together in one data source.
Developers must register by completing a simple form which links to our terms and conditions (http://www.digitalnz.org/developer/api-docs). The current API gives others access to the same functionality we use to build our tools, so is mainly focused on searching the collection and fetching metadata. We will add new features as we need them ourselves, and in response to developer feedback.
We’ve provided a place for developers on the Web site, including documentation (by developers for developers), a starters guide, and a Google Group where developers can share information (http://digitalnz.org/developer).
We’re excited about seeing what gets built using the API, and encourage you to use it too!
Take a Leap of Faith With Us
Our challenge at DigitalNZ is to encourage collecting institutions to see that it’s possible to have a spectacular Web site for digital collections AND to participate in the greater network by sharing information about the contents or digital surrogates.
During our start up phase, we took an ‘ask don’t just take’ approach to harvesting the data and acquiring content, because we wanted to:
- Engage content providers and inform them about the goals of the project so they could work through the problems with us.
- Give organisations the opportunity to open access to their collection content more widely than through a closed ‘portal’ model (initially we weren’t even going to have a ‘DigitalNZ search’ page).
- Educate content providers about new Web concepts so they can make informed choices. The API and RSS feeds, for example, are optional.
- Cache low resolution copies of thumbnails for collection items, and make sure content providers are aware that any risk around IP infringement is their responsibility. The thumbnail cache is optional, but important to improve the performance of our system.
We worked hard to keep the communication channels open and involve our content providers in the process – establishing a blog and information sharing space, ‘being on the end’ of the telephone or e-mail to answer questions (in real time), willingly attending face-to-face meetings, and being open and honest about what we did and didn’t know along the way.
Some of the common questions or talking points we worked through with potential content providers are discussed below.
What’s an API?
Depending on whom we spoke to within the organisation (a curator or a technical representative), the question we found ourselves answering most often was, ‘What’s an API’?
We took time to explain to our content providers what an API is and does, using aids such as this video from TechnoMarketer: http://blip.tv/file/399231/
The difficulty was not having many local examples drawing on publicly-funded or cultural heritage content to refer our content providers to. We’re proud to be among the first to build demonstration applications, and show what is possible when collecting institutions allow open access to information about collection contents.
What’s in it for me?
Having just launched, we aren’t able to quantify all the benefits for our content providers until we see what people start making with the API and our tools.
Yet there are some obvious potential benefits of participation in DigitalNZ, including:
- Tools that will drive people to the unique services of our content providers (tools that can also be used by the content provider. One institution we work with is considering using our search tool as their local search box instead of a Google one).
- Ensuring their visibility in the parts of the Web where their users are and want them – or at least their collections – to be.
- Sharing the development load by enabling others to build applications that surface their New Zealand content.
- A free API that they can use to enhance and add value to their own services.
We try to encourage institutions to think of the real reason we strive to make our collections more accessible on the Web: so that the digital content can be used.
In many cases our content providers are as excited as we are about the possibilities opened up here, and the ability to demonstrate these possibilities to others in their organisation.
An aside: branding
If you look at the screenshots earlier in the paper, you’ll see that there is very little Digital New Zealand branding - except on the Memory Maker, which has branding from one of our partners. We are trying to put ourselves across as enablers, not beneficiaries.
We recognise that our content providers have stakeholders, and funders, local constituencies, and people who hold purse strings who are equal stakeholders in any decisions they make – and are often the very reason that digital content is available on-line.
When Web users discover content such as an image through DigitalNZ, they see only a link. That link leads them directly to the digital object on the content provider’s site.
Some things we suggest our partners consider to maintain their institutional profile in the “Web of data” (Coates, 2005) include:
- Are the URLs for your content objects good ones? Compare http://hdl.handle.net/101382/217 (which has no identification of an institution) with http://www.nzetc.org/tm/scholarly/tei-JCB-031.html.
- Is your institution listed as the ‘publisher’ of the content in the metadata, so that your institution is displayed on DigitalNZ search result pages?
- Is your institution credited as the creator where that is relevant?
- When a person arrives at any part of your site, or your collection object, is it clear who has provided the content on-line, and how you can get back to that institution’s collection or institutional home page?
What if someone does something bad with my data or content?
Our response to this question is simple: ‘What if someone does something really good?’. However, we advise our content providers, ‘If in doubt, leave it out’ – work with safe content first and test the waters until you’re comfortable. If you don’t try at all, however, you’ll never find out.
It’s important to emphasise that we have imposed some limits on what New Zealanders and others are ‘free’ to do with the data and content we make available through DigitalNZ.
- You must not use the DigitalNZ API or the Digital New Zealand content in any context that is obscene, indecent, pornographic or otherwise inappropriate.
- You may not otherwise reproduce, modify, distribute, decompile, disassemble or reverse engineer any portion of the DigitalNZ API or the Digital New Zealand content.
You can read the full terms and conditions on the DigitalNZ Web site at: http://www.digitalnz.org/terms and for DigitalNZ Kete at: http://kete.digitalnz.org/about/topics/show/21-kete-digital-new-zealand-terms-and-conditions.
At the time of writing we are only at the beginning of the journey. Nonetheless, here are a few suggestions that we think will benefit readers.
Develop solutions to fit the environment rather than expect the environment to fit the solution.
We started out hoping that OAI-PMH would be the best way to harvest metadata from content providers. However, very few organisations are set up to do this, and it was clear that we had to work on something easier. So we set up harvesters using XML sitemaps and RSS feeds. More harvesting methods give us greater flexibility – and a side benefit for content providers of having sitemaps for search engines to discover their content.
Similarly, we didn’t develop specs for our API or HTML interface up front, waiting instead to see what sort of data content providers would give us. The lack of standardisation in coverage information (i.e. geographical locators), for example, means this isn’t currently provided as a filter for discovering content.
Remember that the value of your filters, facets, and API is proportional to the quality of your metadata.
Developing a good search experience is not just about being able to harvest metadata. Good quality metadata aids discoverability of content. And good quality metadata is not just about structure, but also about shared approaches to data values in dates, subject terms, and so on. Letting users enhance metadata might allow us to benefit both content providers and users – a test for another day.
DigitalNZ also wants to increase understanding and knowledge about improving metadata quality among content providers. Our good practice guidance includes advice about metadata creation (see http://www.digitalnz.org/contributor/describing/) and, once fully operational, our on-line help service will contribute to this goal.
A lot of metadata exists telling people what they can’t do with content and far less telling people what they can do. Make both obvious.
New Zealand content providers have more to learn about digital rights, permissions, and licenses. Creative Commons licences (New Zealand has its own versions) are not always the right answer, especially in the cultural heritage sector. You can’t license what you don’t hold copyright to.
It can also be difficult for content providers to be certain if an item is out of copyright. We have processes in place for fast take-downs should any issues arise, but are also looking to develop tools to help reduce uncertainty.
Participating in a Web of data also makes it important to keep information consistent across all of the locations where users might encounter access points to collections. It’s confusing for users to be remixing content in one application, only to locate the source on an institution’s Web site and be told they can’t reproduce it without permission.
Provide for education, advice, and examples
Convincing content providers to pre-approve heritage content for change; or set metadata free; is difficult when you can’t predict how the content might be used. We were glad some were willing to take a leap of faith with us.
Part of the barrier was lack of awareness or understanding of new generation Web concepts. For example, content providers often need advice on how to use Creative Commons licenses appropriately before they are willing to adopt them.
We hope by showing people some example tools and applications, we are helping to break this barrier down.
Digital New Zealand is as much about good process and problem solving, and enabling others to find share and use New Zealand content, as it is about delivering digital experiences.
At the time of writing, we are starting to evaluate the test solutions developed in the first stages of the programme, and beginning to identify new problems to solve in the next phase. This iterative approach allows us to develop solutions that are agile and responsive to the fast-changing pace of the digital environment.
Do get in touch if you are interested in what we are doing. We’d love to talk to you about:
- Making New Zealand-related content around the world easier to find, share and use (including in international digital collections)
- Working with institutions that have opened up API data access
- Finding partners to work jointly on innovative software solutions.
We encourage you to make an effort to locate the DigitalNZ Web site (http://digitalnz.org) as we explore and push more boundaries on the ever-growing New Zealand digital space, and welcome your feedback or questions on the work we have done so far.
We hope you will soon see tools and experiences powered by DigitalNZ appearing all over the wide world Web wherever information about New Zealand appears.
The following institutions provide content to DigitalNZ initiatives:
- Archives New Zealand Te Rua Mahara o te Kāwanatanga
- Alexander Turnbull Library, National Library of New Zealand
- Auckland Art Gallery Toi o Tamaki
- Auckland City Council Archives
- Auckland City Libraries
- Auckland University of Technology
- Auckland War Memorial Museum Tamaki Paenga Hira
- Christchurch City Libraries
- Dunedin Public Libraries
- Hamilton City Libraries
- Hocken Collections, University of Otago Library
- Kete Horowhenua
- Kowai Archives
- Lincoln University
- Living Heritage
- Massey University Library
- Ministry for Culture and Heritage
- Ministry of Education
- Ministry of Fisheries
- Museum of New Zealand Te Papa Tongarewa
- New Zealand Electronic Text Centre
- North Otago Museum
- NZ On Screen
- Otago Museum
- Our NZ
- Puke Ariki
- The Nelson Provincial Museum
- The New Zealand Film Archive Ngā Kaitiaki o Ngā Taonga Whitiāhua
- Rotorua Museum of Art & History, Te Whare Taonga O Te Arawa
- The University of Auckland Library
- University of Canterbury Library
- University of Otago
- University of Otago Library
- University of Waikato Library
- Upper Hutt City Library
For information about repositories and content sources included in the DigitalNZ API, see: http://www.digitalnz.org/about/list-of-contributors.
Jones, S. (2008). Get ‘Coming Home’ on your school’s website, wiki or blog, Lunch Box: Software and digital media for learning, 14 November 2008. Consulted 29 January 2009. http://lunchbox.org.nz/2008/11/get-coming-home-on-your-schools-website-wiki-or-blog/
Jones, S. (2008). Liberating our digital content. Lunch Box: Software & digital media for learning, 6 December 2008. Consulted 29 January 2009. http://lunchbox.org.nz/2008/12/liberating-our-digital-content/
Landry, C. (2007). Creativity and the city: Thinking through the steps. The Urban Reinventors.. Consulted 29 January 2009.
National Library of New Zealand (2006). NZ Digital Content Strategy. Last updated1 September 2008. Consulted 29 January 2009. http://www.natlib.govt.nz/about-us/current-initiatives/nz-digital-content-strategy.
New Zealand Government (2007). Creating a Digital New Zealand: New Zealand's Digital Content Strategy, Wellington, N.Z.: National Library of New Zealand.
Singer, A. (2009). 49 Amazing Social Media, Web 2.0 And Internet Stats. The Future Buzz. Last updated January 22 2009, consulted 29 January, 2009. http://thefuturebuzz.com/2009/01/12/social-media-web-20-internet-numbers-stats/