April 15-18, 2009
Indianapolis, Indiana, USA

Workshops: Description

Evaluation Methods: A Practicum

Kate Haley Goldman, Institute for Learning Innovation, USA
Jes Koepfler, Institute for Learning Innovation, USA

So what can evaluation really do for me? And how do I actually evaluate new media tools and projects? Traditional evaluation models and techniques do not always make sense in the context of evolving new mediums. Once you even know why you are evaluating, it is still another matter to actually know which methods are appropriate. Much of the time, technology evaluation focuses on whether or not the tool is usable and never discovers what visitors gain from using it.

This workshop will cover the full breadth of evaluation components, methods, and applications so that workshop participants leave with the resources and understanding necessary to effectively test out a new tool, or plan an evaluation for any technology project at their institution or organization.

From this workshop, participants will:

  • Understand why and when they should evaluate
  • Relate evaluation to their current project needs
  • Become more aware of the breadth of evaluation methods and know when to apply them
  • Have access to a wiki of resources for evaluation and a bibliography of literature for future review

What is evaluation?

Evaluation is both a process and a product. The process is applied research with an end-product of data that will help you make informed decisions about your project, program, or exhibition. Evaluation brings the visitor's voice - an easily overlooked stakeholder - into the design process to ensure that usability and learning goals are achieved from the end-user perspective. Your evaluation plan will look different based on the needs of your stakeholders, the goals of the project, and the function of your piece. In this workshop, we'll step you through the sequence and scenarios to answer these key questions and get you to an informed and reflective design practice.

What are museum technologies?

For the purposes of this workshop, we will consider 'museum technologies' broadly as any technology that is employed within a museum context, whether on-site, online, or in the community, with informal learning goals at the heart of its purpose. Such technologies might include interactive kiosks, video games, flash animations, films, websites, online scientist interviews, podcasting, Moodle, virtual worlds, public radio documentaries, streaming video, distance learning simulations, mobile devices, blogs and wikis, open source applications, Google gadgets and widgets, Twitter, and.... You get the idea.

What will we do in this workshop?

Through a combination of presentation, demonstration, large group discussion, and small-group scenario practice, we will walk participants through the basic evaluation steps from beginning (planning and front end), to middle (formative usability and narrative flow), to end (remedial and summative). We will provide participants with a toolbox of evaluation strategies and guidelines for when to use them, including but not limited to logic modeling, personal meaning maps, paper prototyping, think alouds, focus groups, online surveys, heat mapping, remote user testing, observation, website analytics, and more. We will provide ample time for a Q&A session for participants to present their project goals as a scenario for evaluation planning. We will also suggest both proprietary and open source software solutions for employing some of the evaluation methods we will cover. The workshop aims to be lively and interactive with a real focus on translating theory into practice and being as relevant to the specific needs of the workshop participants as possible.

This workshop intends to cover a lot of ground in a short amount of time, relying on a wiki of resources, applied scenarios, and a bibliography of literature for participants to access for greater depth into any one aspect of evaluation or application of a specific methodology afterward. All levels are welcome as we will build the workshop organically based on the needs and experiences of the group.

Participants should bring a laptop to collaborate in building a wiki of resources and to access data being discussed in the group.

Workshop: Evaluation Methods [Afternoon]

Keywords: evaluation, methods