10 Museums, 12 Months, 1 DAMS: Adventures in centralized systems at Balboa Park

Howard Goldstein, HR Goldstein Consulting, USA; Perian Sully, Balboa Park Online Collaborative, USA

Abstract

Digital asset management (DAM) can be difficult, expensive, and time-consuming for a cultural institution to implement, yet DAM has become a central process to everyday workflows within cultural organizations. For ten small and mid-sized museums in San Diego’s Balboa Park, the Balboa Park Online Collaborative (BPOC) undertook a project to rapidly deploy a Digital Asset Management System within the space of twelve months. This paper will provide a blueprint by outlining the process and steps BPOC took to analyze user needs, choose an application vendor, and implement the system, with special focus on the opportunities and challenges of providing centralized solutions to institutions with localized or parallel interests. This model can be of use to large organizations with disparate departments, colleges and universities, and collaborating organizations.

Keywords: digital asset management, DAM, centralized IT, cloud computing, BPOC, Balboa Park, collaboration, rapid deployment

Digital asset management (DAM) has become a core business process within cultural institutions. DAM systems (DAMS) house, manage, and facilitate a workflow for use of collection images, event images, conservation images, PDF files of reports, and more recently, video and audio files.

For those unfamiliar with DAM, digital asset management is a series of workflows that utilize software and hardware to organize digital files, commonly referred to as “assets,” which include images, video, audio, documents, etc.  DAM systems comprise the software and hardware packages that manage the workflow. The software can automate the process of creating derivative versions of the original files as needed, for use in specific instances like a web site, PowerPoint presentation, etc. without the user needing to know how to do this or have the software on their desktop computer.  This process also protects the original and eliminates the need for multiple versions of the original. For cultural organizations, a body of digital materials often includes representations of collection objects, and content from events, exhibits, marketing materials, and architecture.  Used for printed and online scholarly and educational use, as well as marketing publications, the assets are used every day by museum staff to further the mission of the institution to educate and enlighten, and to establish the brand of the organization to the world.

The Balboa Park Online Collaborative (BPOC) addresses common technological problems among 27 arts and culture organizations within San Diego’s Balboa Park. BPOC assists organizations with IT, website and mobile development, network infrastructure, digitization, and online collections access. (Cherry, 2010) BPOC’s collaborative model has allowed the member organizations to share resources and knowledge with one another, often through BPOC, resulting in increased partnership and collaboration.

The project outlined in this paper has been successful, in part, due to the physical location proximity and context that the Balboa Park organizations enjoy, and the strong leadership and strategy of the BPOC.  This is a somewhat unique arrangement.  The real question is how can other organizations take advantage of this model?  University cultural heritage institutions would be one example.  Many universities have various collection libraries as well as museums within their orbit. They usually have a strong centralized IT infrastructure for the university itself, which could be leveraged in a similar way that BPOC operates.  This is the case at Yale University, who manages a central DAMS through their Office of Digital Assets and Infrastructure, that serves a number of their cultural organizations. A second example might be small to medium sized cultural heritage institutions located within a geographic area like a city, which could band together to implement a similar strategy.  Institutions united by a topical reference could also find common needs addressed. Large organizations with disparate departments would also benefit from this model.

The collaborative model was extended for the Balboa Park DAM project; by leveraging technological resources within BPOC staff and asset storage in its private cloud, organizations were offered the structure to overcome the usual problems of DAM implementation for small to medium institutions - lack of budget and staff, and little or no IT infrastructure. The decision was made to utilize this structure to implement a DAMS in an innovative way. Rather than installing individual DAMS software for each institution, BPOC would make one system available to many institutions with discrete directories for their assets and an institution-specific security structure for their users. This would allow BPOC to take advantage of economies of scale for hardware and software, efficiency of centralized maintenance, and the opportunity for collaboration between institutions via a common database structure. The overall budget for this project was $300,000, which covered hardware, software, consulting fees, and salaries.

In January and February of 2011, BPOC staff, along with author Goldstein as consultant, undertook a two-month review of DAM systems and user assessments, and chose Piction, an Australia-based company whose clients include the Queensland Government and the Cleveland Art Museum. (http://www.piction.com) Three Balboa Park organizations were chosen for a pilot project - the San Diego Air and Space Museum (SDASM), the Museum of Photography (MOPA), and the San Diego Natural History Museum (SDNHM). Two other museums, the Mingei International Museum and the San Diego Museum of Art (SDMA) requested to join the pilot project and they were added to the project following the initial evaluative phase. These organizations were selected based upon the quality of their assets, the diversity of their collection management systems and metadata export formats, and the willingness of the organization to be a testing ground for this model. The plan was to roll out DAM capabilities to these three organizations by the end of 2011 and add 7 more park organizations by the end of April, 2012.  The project tasks and workflow, and lessons learned from the pilot implementation would serve as the basis for the next phase of rollout of the system.

The notion of DAM implementation for any institution is daunting. The software and hardware costs are high. To date, successful projects have been determined by collaboration with a strong vendor, and/or a dedicated staff member assigned as coordinator of the DAMS from the start. The majority of institutions in Balboa Park have little or no internal IT infrastructure. Many of the institutions would be considered small to medium in regards to their collections. It would be nearly impossible to justify the hardware and software costs for any of the individual institutions that make up the park, and the support structures to implement and sustain the projects would be very difficult to establish.

DAM implementations often become multi-year projects for good reasons: 

• The organization of the assets is a huge undertaking. De-duplication is generally the most time consuming task. Assessing file naming conventions and quality of legacy files is an important part of the process.
• Establishment of a standard for asset file metadata can be time-consuming. Different types of assets will require different schemas. The metadata almost always does not already exist and a vehicle for data entry must be decided on.
• Decisions on viewing and use permissions for the assets can become very complicated. Rights and reproduction restrictions play a large role in these decisions.  To make optimal use of the DAM workflow, automated derivative creation should be implemented.
• Successfully utilizing a DAM workflow requires a process mind change from staff.  This is usually the hardest aspect to control.

The pilot organizations differed somewhat in their level of preparedness and organization for the aforementioned tasks. However, the organizations shared many issues, including scattered directories of assets, little or no metadata attached to assets, worries about levels of security for viewing and use, and the need to hopefully migrate collection information or library system metadata to a DAMS as a minimal schema to search assets. The most difficult aspect of these issues involved this last one; each of the organizations used a different collection information or library system.

The general similarity of need among the organizations made the concept of the project enticing: one overarching software system with individual sub-instances to house digital assets and facilitate a new workflow; and one hardware platform, managed centrally with the necessary resources and business processes. Co-operation by the vendor in adjusting their normal license structure allowed the BPOC to realize a financial advantage to this arrangement. The existing IT infrastructure of the BPOC created a facility for implementing a web-based access application with asset storage in the background. The differences between the individual organizations’ instances would revolve around their user access strategies, their metadata schemas, and migration of metadata from their respective collection information and library systems. The basic structure of the implementation presents little difference to that of a single institution.

Unlike many other institutions who have instituted DAMS, the timeline for implementation was significantly compressed; funding for the project came, in part, from a two-year Institute of Museum and Library Services National Leadership Grant for “Presenting Balboa Park,” an online portal providing cross-institutional and public access to museum image collections. (BPOC, 2009) Development of Presenting Balboa Park and an early launch of the website in April, 2012 required expedited selection and implementation processes.

As a result, BPOC bypassed the traditional RFP and comprehensive needs assessment process. It created a short list of DAM system vendors and invited them to demonstrate their products for key staff at the pilot organizations. BPOC and Goldstein also interviewed these staff to identify their current digital holdings, usage, workflows, and concerns. Based upon these interviews, BPOC developed a set of initial user security profiles and transformation formats that would satisfy a majority of staff upon adoption within the organization. Further adjustments to the user roles would be adopted as staff becomes more comfortable with the system and needs are further defined.

Table 1: Basic security profiles with permitted derivative access

Following the demonstrations and interviews, all pilot stakeholders met to discuss their impressions and questions. The consensus was that Piction appeared to be the most technologically-capable to support a collaborative project of this sort, and the enthusiasm of the Piction staff to engage with Balboa Park for this project were important factors influencing the decision by the organizations. Cost, of course, was another reason for the final decision. BPOC was able to offer to the partner organizations two years of no additional licensing fees, and cost sharing would occur between partner organizations after the expiration date. Additionally, BPOC also recognized that it would be able to use Piction as the backbone to support the Presenting Balboa Park project, ensuring that much of the anticipated programming costs for the project would be folded into the infrastructure that supports a shared DAMS.

Figure1: DAM implementation structural diagram

Infrastructure and Hardware

BPOC is able to provide economies of scale and staff assistance through centralization of computing resources, network services and IT support. BPOC has implemented a private cloud infrastructure for a number of technology initiatives that are subscribed to by member institutions, including organization websites, backup storage, and a Park-wide online calendar. (BPOC website, http://www.balboapark.org/bpoc/work/infrastructure) The DAM project builds upon these successes by utilizing the professional and technical resources at BPOC’s disposal. Just prior to installation of Piction, a 1GB network was installed throughout the Park, providing each organization access to a 50mb connection to the Internet. (Cherry, 2011) Consistent, high-speed connectivity to the network is crucial for a shared DAMS project. Despite high-speed Internet connectivity in the park, the inconsistent quality of individual firewalls, servers, storage space, and wiring meant that it would be more efficient for Piction to access the asset server at BPOC, instead of connecting to individual organizations’ repositories.

The hardware selected for the project was easily attainable and configured and cost around $15,000. BPOC’s IT staff worked with the Piction staff to define the best method to organize processing power, hard drives and backup. Two physical servers form the “brains”” the asset server and the application server.

The asset server, where the high-quality access files are located, is a 24-disk storage array comprised of 2TB drives, for 48TB of storage. Twelve of the drives form a RAID6 array, and another ten drives form a second RAID6 array. RAID arrays are necessary for protecting the assets in case of hard drive failures. Two drives are reserved for global hot spares – unused hard drives that are kept in standby mode, which become active if a disk fails. The operating system is a Linux variant called Open-E DSS6. This is running on High Performance CompactFlash drives for increased protection. There are an additional two drives on standby should a failure occur.  The asset server also contains two power supplies and dual-boot ROMs on the motherboard.

The Piction software runs on a separate application server that manages the databases and asset requests by users. It is running a free version of VMWare ESXI 4 on Windows Server 2008R2, and the server itself is a Quad-core Xeon CPU with 24GB of dedicated RAM. The server has eight 1TB SAS hard drives, five of which are configured in RAID5, two in RAID1, and one reserved as a global hot spare. It also has two power supplies and dual-boot ROMs on the motherboard. Piction itself runs an Oracle database that also supports communication with other SQL-compliant databases.

Organization users log in with their individual credentials through a web browser and see a start page with their branding.  Users only see the asset archive for their institution’s content.  The specifics of what assets they see and have access to are determined by the designated image administrator who decides upon their user profile and permissions.  This separation is accomplished by virtual sub-“installations” (called instances) of the main Piction installation, though one central back-end database exists to organize the instances. The decision was made to utilize the internal user security model of the Piction application rather than linking any organizations’ individual Active Directory model.  This allowed us to create a denser and more specific permissions model for the targeted users of the DAM system.

Asset Coordination

Prior to launch of the project, all of the Balboa Park organizations had existing digital assets housed on their internal servers. Many museums have been creating digital content for over a decade, and, as is often the case, assets were stored on individual desktops and shared directories. BPOC determined that it wanted to focus on loading the Piction instances with permanent collections images and videos first; these were selected for the following reasons:

• Likelihood of available metadata ensured that retrieving images would show the potential benefits of using a DAM in all organization departments.
• Collection assets are usually housed in one central directory, and so would be easy to transfer into the asset server.
• File names are generally somewhat standardized to coincide with accession numbers.
• Images of collection items are used by most departments within the museum, so as staff become used to using the DAMS to retrieve those images, they may begin to see how it can work for their department.
• The Presenting Balboa Park project, as explained above, is collections-centric and BPOC itself was founded with the goal of making collections accessible to the public.

Since early 2010, a massive digitization effort has been underway at BPOC, and by April, 2011, nearly 150,000 photographic collection and rare book items from ten organizations had been captured using rapid digitization stations. Another 1200 videos had been digitized through BPOC as well. These captures had been backed up on the asset server, and so BPOC had a large repository of images and video with which to start.

Figure 2: Screen capture of one of two sets of asset directories, dated January 24, 2012. This screenshot illustrates approximately half of the total assets requiring management. 

For the pilot organizations, BPOC took copies of all available assets and placed them in an organizational directory for evaluation, prior to incorporating into the final repository with the approved assets. Over the years, without strict DAM practices, it is common for staff to move, copy, and create multiple resolutions of files for their use. This is understandable, but it does necessitate that, prior to ingestion into a DAMS, an exhaustive file cleanup process must happen to remove redundancies and damaged files.

Files were initially gathered from the participating institutions via transfer onto external hard drives, but organizations are now able to access the repository via SFTP. Archival masters were segregated and moved onto segregated storage drives. For use by Piction, high-resolution access files were preferred for retention, but if only one image of the object existed in a low quality JPEG, it was retained in the asset server.

After the collection of the assets, the cleanup process began. Author Sully used a combination of three software programs to perform the work:

• BeyondCompare Pro – Used to compare directory contents and identify duplicate files. Offers features that allow for direct comparison, regardless of file extension, and to see the size and content of the file before determining if it should be deleted. Allows for deletion of files in bulk. $50.00 for a single-user license. (http://www.scootersoftware.com/)
• ReNamer – For batch renaming of files. User can set up a complex stack of rules that allow multiple types of filename problems to be fixed simultaneously. Freeware, Windows. (http://www.den4b.com/?x=products)
• Adobe Bridge CS5 – Sometimes it is necessary to view a number of images side-by-side before determining what may be safely deleted. It can also be useful for seeing corrupted files that appear as if part of the image is covered in black. It is also used to embed descriptive metadata into the IPTC Core fields. (http://www.adobe.com/products/bridge.html)

Cleanup was performed locally at BPOC. This not only ensured that the assets loaded into Piction were of the required quality and filename standards, it also provided a failsafe in case of accidental deletion and the process didn’t impact the day-to-day activities of the staff. Sully coordinated these activities with key players within the organizations and worked with them to align these changes to their daily workflows and procedures. For medium-sized collections with more than 100,000 digital assets, it required approximately 100 hours over three to four weeks to complete the process.

Filenames for collection images were standardized to align with metadata exports from EmbARK and PastPerfect collection information systems. The filenames would reference the accession number of the object, and hyphens were preferred over periods within the string. For alternate views, an underscore would delineate the additional information. For example, three photographs of object 2009.01.024 might have filenames 2009-01-024.TIF, 2009-01-024_det1.TIF, and 2009-01-024_det2.JPG. This has two major advantages. First, it makes it easy for the evaluator to sort and compare images when deciding what to retain. Second, it makes it easy for the metadata ingestion process to recognize the related Accession Number string within the filename and match it to the object metadata.

Redundant and complicated directory structures are removed as well. Prior to cleanup, one organization with a collection of 25,000 catalogued objects had 154,336 assets within 2,465 subdirectories. Following cleanup, the total count was 49,862 assets in 721 subdirectories. Without immediate direct access to the staff, the curatorial and exhibitions working directories remained mostly untouched, other than deleting redundancies. Collections directories were reduced to twenty-two, organized by decade.

Loading

Once the assets are prepared and moved into the dedicated directory, it is simple to load them in batch within the DAMS. The instance is prepared (a process that takes approximately 10 minutes within the Piction Administrator panel) and the system is pointed to the directory. It can recreate the directory structure automatically within a protected section called a Sub-Category. Access to this Sub-Category is restricted to the Image Manager and Super User roles, and those staff can move the directories into the General Access Sub-Directory when they determine that the assets are appropriate for use by other departments.

After the assets are loaded into the system, the metadata is exported from the collection information system and mapped to metadata fields created within the Administrator interface. Selection of the fields is determined by the organizations, and a data dictionary or metadata map for each organization is helpful here. The map should outline the following:

• Public Field Display – this may be either the exact name of the field (ex. “Accession Number” for a field that exports as “Object ID”), or “NO” for those fields which should be present for staff use, but not for the public. This field is unnecessary for DAMS implementations that won’t interact at all with the public, but it can be of great use for other collection online efforts.
• DAMS Field – used for listing all fields that will be exported for mapping into the DAMS.
• Allowed Roles – for assigning which users have access to specific fields.
• Collection Information System (CIS) Field Name – the plain-text name of the field within the collection information system as represented to the user (ex. “Accession Number,” “Artist Name,” “Creation Date,” etc.)
• CIS Internal Field Name – the internal code that the CIS uses for a field. For example, the field the user sees as “Accession Number” may be coded in the system database as the “Disp_Access_No” field.
• Table Name – for certain relational databases, information is organized into discrete sets of information. Object information may belong to a set called “Object”, but the artist information will belong to “Creator.” It’s helpful to map this before determining how to write the CIS export process.

Table 2: Examples of data dictionary rules. In this example, the access roles (General, Super User, and Image Manager) are being mapped to the viewable fields, prior to export from a CIS

Deployment

Once the assets are ingested, the metadata linked to them, and the user accounts created, the login to the interface accounts are sent to the Image Manager at the organization, who then provides the login information to the users. BPOC organizes a one-day training for them, and offers ongoing training sessions. BPOC also centralizes support and troubleshooting and provides feedback and requests for adjustments to the vendor.

At the institution, the Image Manager is responsible for evaluating how the system is used, advocating for its adoption by other departments, cleaning up discrepancies, updating metadata through the client interface, answering direct questions by fellow staff, and approving image requests.

Current status

As new collection items are imaged or videos digitized, assets are copied by museum staff via SFTP into dedicated directories on the asset server. Piction runs a nightly process to ingest these materials into the system.

The table below shows where we are today. It reflects that the load process is performed in batches, dependent upon the types of files available within the asset server (image, video, audio, document). It also shows the outstanding number of assets that have been moved to the server and not yet loaded into Piction. There is also an expectation that, as other departments adopt the DAM workflows, there will be a large number of non-collection assets to come, and new security profiles for individual departments will be required.

Table 3: Number of Total Assets and Amount in Piction as of March 13, 2012

The image assets for MOPA, Mingei, and SDMA constitute almost all of their collection images. The available materials for SDNHM are currently a very small percentage of available digital assets; a more extensive institution-wide asset inventory is currently underway. SDASM, with a large physical photographic collection of over one million items, has slightly more than half of their available digital assets in the DAMS.

Five more organizations will be deployed by the end of May, 2012, with an additional four by the end of September, 2012. These organizations are the Timken Museum of Art, Museum of Man, Reuben H. Fleet Science Center, San Diego Floral Association, San Diego Hall of Champions, San Diego Junior Theater, San Diego Model Railroad Museum and BPOC itself.

The key fact to glean from the above table is that the current total amount of assets in the system would be considered a medium implementation for one institution within the cultural heritage arena.  The expected amount of assets for these five organizations would still be considered of medium size.  Extrapolating to the inclusion of all the park’s organizations would probably put the DAM implementation on the level of a large institution like the Metropolitan Museum of Art in New York. We can easily realize the cost and resource savings immediately from a centralized implementation as compared to each of the above organizations trying to implement their own system with the inherent hardware and software, IT resources, and internal staff resources. This will only become more apparent as the system is scaled up. 

The DAMS was up and running at a level slightly above basic functionality for the pilot organizations within about eight months. The time frame to bring the seven or so subsequent organizations online will be in the order of half that time. Using the pilot program as a model, we will have been able to bring ten organizations online with DAM capabilities in the time most of the industry takes for one organization.

The rapid-deployment model used for this project need not apply solely to DAMS implementations.  There are many software and hardware workflows, processes, and functionalities that could be realized through a rapid-deployment structure.  With the reality of today’s scarcity of all kinds of resources, coupled with the explosion of demand and need for all kinds and quantities of data in the cultural heritage sector, we need to think out of the box and utilize innovative technologies to their fullest.

Various levels of technology and philosophy exist today and continue to develop that would support this type of implementation. On the hardware side, faster bandwidth, external hosting, cloud computing, virtualization of hardware, and the continual lowering of costs for these capabilities sets the stage for the realization of a robust, centralized IT situation for small to medium organizations and a collection of multiple organizations. On the philosophy side, the recognition and momentum for further collaboration between organizations to enhance the experience for scholars and the public is a strong influencer. Organizations with holdings by the same artist, or objects that have topical similarity, can present their content in tandem. The possibility of connections between the content of even seemingly dissimilar organizations is enticing. 

Acknowledgements

No project of this scale is the result of a single person. Thus, Perian would like to acknowledge the efforts and feedback from the staff at the Balboa Park organizations, as without their trust and commitment this would not be possible. She would also like to recognize Howard Goldstein for his guidance throughout the project. Special mention must be made of the attention, care, and attitude (and, it must be said, tasty beverages) by Piction CEO Erick Kendrick and the Piction team. And finally, the rest of the BPOC staff who have invested considerable time and knowledge to making this project a success: Rich Cherry, Christina DePaolo, Scott Granger, Jason Quinn, Christopher Borkowski, and Madison Shelestak. Howard would like to thank Perian and the entire staff at BPOC and their organizations for their welcoming environment and professional demeanor, and Erick Kendrick and Adam LaPorta for their commitment to this project and the fun times! He would especially like to thank Rich Cherry for including him in an exciting and innovative project. Lastly he would like to thank his wife for putting up with his travel especially during last winter’s snow storms.

References

Balboa Park Online Collaborative (2009) Balboa Park Online Collaborative Awarded $675,000 National Leadership Grant, 2009. Consulted February 26, 2012. http://www.balboapark.org/bpoc/blog/balboa-park-online-collaborative-awarded-675000-national-leadership-grant

Cherry (2010). Cherry, R., A Walk in the Park: The Balboa Park Online Collaborative First Year Report. In J. Trant and D. Bearman (eds). Museums and the Web 2010: Proceedings. Toronto: Archives & Museum Informatics. Published March 31, 2010. Consulted January 26, 2012. http://www.archimuse.com/mw2010/papers/cherry/cherry.html

Cherry (2011). Cherry, R., Collaboration, Dark Fiber and the Challenges of Deploying Mobile Infrastructure. In J. Trant and D. Bearman (eds). Museums and the Web 2011: Proceedings. Toronto: Archives & Museum Informatics. Published March 31, 2011. Consulted January 26, 2012. http://conference.archimuse.com/mw2011/papers/collaboration_dark_fiber