Museums and the Web

An annual conference exploring the social, cultural, design, technological, economic, and organizational issues of culture, science and heritage on-line.

You are hereMW / What you thought of MW2009. Evaluation Data Analysis

What you thought of MW2009. Evaluation Data Analysis

Thanks so much for your evaluations of MW2009 – if you haven’t completed one, you still can online at Our preliminary analysis of these is proving immensely useful as always, but we’re always happy to hear more (so respond online even if you filled one out at the meeting but have had additional thoughts since).

You don't always agree ...

We’ve learned over the years that there is nothing that occurs at Museums and the Web that does not have at least one person praising it – and someone who thinks it’s terrible! This year was no exception: Max Anderson’s plenary address received reviews of “Fabulous speaker. I could listen to him for hours.” and “exceptional, set great tone” as well as “not relevant” and “I did not see connection with rest of the session”. The unconferences were greeted as “great opportunity for informal discussions” and the venue of a “life changing conversation” to “more of a bitch session” and “completely unproductive”. Fortunately we understand that the range of the MW community is one of its strengths and while we carefully assess each comment to see if we could better satisfy that individual, we listen to the aggregated data before changing something we know works for many. So we’ll report some hard numbers, and some impressions.

Things are pretty good
The MW evaluations rank everything from 1 to 5 with 1 as ‘Exceptional’ or ‘Completely Satisfied’ and 5 as ‘Disappointing’ or ‘Unsatisfied’. Fortunately, 44% of you ranked Max Anderson’s opening plenary 1 out of 5 and another 39% thought it ranked a 2 (83% gave it a superior rating!), so it seems we got off to an excellent start. The next highest ratings of 1 were given to the unconference session – an innovation this year. 42% of the respondents gave these a 1, and another 27% ranked them 2 (for a total of 69% superior). The Program Committee need not retire however, because although the regular sessions received only 28% 1 rankings, they got an overwhelming 51% 2 rankings for a total of 79% superior – exceeding even the positive response to unconference sessions by fully 10%. And the Program Committee selected mini-workshops received an overall 74% superior ratings too.

You like social events

I’m pleased that the evening social events were well received; after all they are expensive, involve a great deal of extra effort and we include them in everyone’s registration, so it would be a shame if they weren’t enjoyed. 73% of attendees thought they were superior, and with only 2% reporting that they enjoyed them only somewhat, or not at all, they had the lowest lever of disappointment of any activity at the conference. By comparison, 14% of attendees reported being somewhat or not at all satisfied with the unconferences.

Breakfast will always be too early for some people ...
This year the Birds of a Feather breakfasts included a full hot breakfast with lots of great options including granola and yogurt for the grainy ones among us, but all the extra food didn’t attract any greater approval. Some people simply won’t get up for breakfast regardless, leaving 8% only somewhat or not at all happy. The 52% of respondents who ranked these as 1 or 2 probably included most of those who actually came (we set breakfast for 240 but even then many seats were untaken, so we estimate that only about 40% of attendees participated).

Small but devoted following for ...
Every year there are many types of sessions that receive few rankings but those who attend often report that they liked them best of all of what occurs at the meeting. For example, the Crit Room and Usability labs were only ranked by 39 and 29 attendees respectively, but the Crit Room was reviewed as “painful to be dissected but good” and “love this format” while the Usability Lab received comments :”Sooo useful” and “loved the dynamics and usefulness of this”. Few attendees can come to the pre-conference tours on Tuesday – they are strictly limited in numbers and occur two days before the opening of the meeting – but every year some of those who do rank this as their favorite activity of the whole meeting.

The Exhibit Hall is an opportunity missed by many
My disappointment each year is the way that MW attendees view the exhibit hall; this year was no exception. For every “very cool” and “pretty crowded but lots of opportunity to visit” there were several “nothing relevant to me”, “don’t have any $” and “not enough time for this”. Add that to “some companies do a bit too much marketing” and “same stuff year after year” and I think we have some very big misunderstandings about what the exhibit hall is and how to benefit from it. In my view, the companies that come to Museums and the Web are implementing some of the most exciting things on the Web every year. They include people with far more specialized expertise in many areas of Web development than almost any institution attending MW can afford to have on its staff. And they are there for 12 full hours for attendees to interrogate for free. The learning opportunities that the exhibits present are not only relevant to those who have budget this year to hire one of these firms; everyone can take advantage of them. Thank goodness some companies are still there, year after year, but these products and the interactives developed by the design firms, are definitely not “the same stuff year and year” but rather are evolving and improving based on requirements expressed by other museums and opportunities presented by new technologies (typically exploited earlier by the vendors)– there is a huge amount to be gained by talking with the exhibitors. Yet MW attendees more often than not fail to engage them; from responses on the evaluations we suspect some don’t even realize the exhibitors are there all day Friday and are too busy at the reception on Thursday to bother!

Demonstrations perennial favourite
Fortunately, the demonstrations generally attract a lively crowd and those who attend enjoy them fully. We get a few “a bit dull” and “too noisy” but largely it’s “discovered a few gems” and “a great opportunity to experience others' projects”. Statistically, 68% gave these a 1 or 2 ranking, making them as popular as the unconferences and with a larger number of respondents (fewer passed the demos by altogether).

New Ideas

Every year we get a few good ideas directly from participants. My favorite this year was “a session in which we have 5 minutes to demonstrate an idea”. We’ve played with the ‘pitch’ format previously – I think next year we may give it a try. We also get a few impossible suggestions each year – this year my favorite was to have slides from presenters on the MW website before the conference (and I thought getting papers there was heroic!)

Hard Choices
Finally, each year we hear from numerous people that what they liked least about MW was ‘having to choose between sessions’, ‘having too many parallel sessions’, and ‘staggered start times for sessions’, all of which we won’t cure because we’d hear more complaints if there wasn’t anything you wanted to do:-)

You get back what you put in
In conclusion, this year’s evaluations reinforced something I had long suspected – those who put most into then meeting up front, gain the most from the conference in the end. Or perhaps, those who invested up front came to the right meeting while some others made an error in choosing MW for not having done enough research. In either case, of attendees who reported reading mw online blogs, only 2% reported that MW met only some of their expectations. The rest of those who reported, said it ‘exceeded’ or ‘met’ their expectations.  By comparison, 8% of those who read some conference papers in advance, but did not read mw online blogs, reported that MW met only some of their expectations. While 10% of those who read neither papers nor blogs said MW met only some of their expectations and a further 8% were ‘disappointed’. Although having only 18% of any group less than very pleased may not be bad, the difference between those who made an effort in advance to become more fully acquainted with the meeting and those who didn’t remains striking.

It’s never too late to get more out of the meeting – read a few papers that you didn’t get to hear, visit the conference online at and review some blog posts from the news feeds or some twitter posted urls, or review the slides most speakers have now made available on slideshare that are also linked to their abstracts on the conference site. Think about what you might want to propose for Denver in 2010 – the call for proposals (PDF) is available and the form will be on-line from late summer through September 30, 2008.

jtrant's picture

if you'd like to see the anonymized numeric results from the evaluation form, question by question, we've shared them on-line.


j. trant archives & museum informatics

j. trant co-founder Museums and the Web | partner archives & museum informatics