Enter your email address:

Delivered by FeedBurner

About This Blog

Josh’s Tweet Stream

  • More tweets

« Some fun from Alberta | Main | Barack Obama, will you tap the groundswell? »

October 30, 2008

Forrester Groundswell Awards: How we picked the winners

by Josh Bernoff

Since the competition was so intense this year for the Forrester Groundswell Award winners, I wanted to create as much transparency as I could around the judging process. This is in response to questions from some of the entrants, who clearly put an enormous amount of effort into their submissions.

Q. What exactly was the judging process?

Three people at Forrester were involved in the judging: Zach Hofer-Shall (a researcher on our interactive marketing team), Jeremiah Owyang, and me. We took a first pass to separate the most promising applications in each category, then we spent more time on those promising applications, looking for those which we thought were good enough to win. For an application to win, Jeremiah and I had to agree it was the best in the category. We were looking for both innovation and proof of business (or non-profit) results -- the winners in each category were excellent on both counts.

Q. It looks like some of the entries ended up in a different category from where they were submitted.

In working with clients, we have found that deciding on objectives is often difficult. Successful applications often start off meeting one objective, and end up delivering on others.

In this process, we looked at each application and asked, in what category would this application be most likely to win? Usually, but not always this was the category in which it was entered. But we made some changes as well. In the case of the Brooklyn Museum, there were three entries in different categories, but taken together we thought they demonstrated excellence in the Social Impact category, which is the category in which they won. "In the Motherhood" by LiveWorld was submitted for Supporting, but we thought it was an excellent Embracing application, since the actors in it acted out scripts suggested by the groundswell, and that's why it was a finalist in the Embracing category.

Just like last year, there was one company that stood out for its transformation. We felt this year's entries from Intuit were best recognized together, rather than as separate applications, which is why we gave Intuit the Company Transformation award. I don't know if we could make this a regular category, it just seemed to emerge from the other entries.

Q. How much weight did you give the evaluations from the community?

We were pleased at how many entrants sent their followers to our site, and the reviews that were generated. We all have to acknowledge, though, that the people coming to the site through that kind of promotion were much more likely to say very positive things. Partly because of the potential for this sort of gaming, we weren't going to pick the winners based on a popular vote.

However, many of the comments added insights that we might not have noticed otherwise. So the main value of the reviews, from my perspective as a judge, is that they helped to enlighten me. Taken together with all the other information in the entries and the sites they linked to, this gave us a clearer picture of what was outstanding (or in some cases, flawed) about the entries.

Q. Does this process favor large companies?

Absolutely not. It favors excellence. A small company can deliver outsized results with a small budget, as the Common Wealth Credit Union of Alberta did, and as the Brooklyn Museum did. National Instruments was a medium sized company with an incredible Supporting application. On the other hand, big companies and their agencies or vendors, like Hershey's with House Party, Mattel with Communispace, Starbucks with Salesforce Ideas, and Accenture deserve to be recognized for excellent work, too. It's not about how much you spent; it's about how effective and creative you were.

Q. What exactly does it mean to be a "finalist"?

Finalists are basically applications that could have won, except that somebody else was even better in the category. The 44 finalists this year were a lot, but they reflect the number of excellent entries. However, there's a clear distinction; the winners deserve to be recognized as the best of an excellent group.

Q. Will you do this again? What will you change?

We will definitely do the awards again in 2009. One thing we changed from last year -- making all the entires publicly available -- was a big plus, I expect that to continue. I expect us to make changes based on what worked well and what didn't this year; for example, I hope to start the process a little sooner to give people a longer time to prepare entries and vote on them. I welcome your suggestions.







TrackBack

TrackBack URL for this entry:
http://www.typepad.com/services/trackback/6a00d8341c50bf53ef010535cac872970c

Listed below are links to weblogs that reference Forrester Groundswell Awards: How we picked the winners:

Comments

John Caddell

Josh, thank you for elaborating on the process used to judge the Groundswell Awards competition. It helped me understand better how the judging was done.

I believe shifting entrants from one category to another was a mistake. However well-intentioned this was, it felt to this one entrant as if the rules were changed midstream.

I am a bit shocked, and bummed out, by this statement: "We were pleased at how many entrants sent their followers to our site, and the reviews that were generated. We all have to acknowledge, though, that the people coming to the site through that kind of promotion were much more likely to say very positive things. Partly because of the potential for this sort of gaming, we weren't going to pick the winners based on a popular vote."

I don't believe that soliticing reviews is gaming the competition. As an author of "Groundswell," you know well that most people don't take time to contribute to public sites. While reviewers might be inclined to say positive things, they still had to invest time to visit and rate, and to build articulate arguments. You yourself said the comments were insightful. As a result, I strongly believe the number of reviews was a pertinent factor that should have been considered.

Many entries (including many winners) had no user comments or ratings at all. To me, this was anti-Groundswell and should have been a significant negative factor in the judging. But, clearly, it was not.

regards, John

Josh Bernoff

@John Caddell Thanks for your comments.

Regarding the reviews, if a community says "Hey, everybody in this community who loves us, go rate us at the Forrester Groundswell Awards" then clearly they'll get a lot of positive reviews. It's not that I don't find those reviews valid, but I can't take them as representative of quality. So we read them, but their presence doesn't make that entry win -- and an absence of reviews doesn't make an entry lose. If thousands of people had found our site and were voting, I'd find the majority rule a little more persuasive, but they didn't.

Regarding changing the categories, basically, I don't want to penalize somebody who has a great application if they didn't choose the best category.

I appreciate what you said about changing the rules. We didn't change the rules, in fact we did the same thing last year, but we DID fail to clarify that we do things this way. I apologize for not being explicit about this up front -- that was a mistake.

The comments to this entry are closed.