Capture the Wisdom of the Crowd with Pairwise Voting

By: Hutch Carpenter | November 7th, 2012

Gathering and channeling community feedback is a critical element in the crowdsourcing process. The objective is to combine each individual’s slice of the “truth” – knowledge, values, decision frameworks – into a collective wisdom about highest potential ideas. As James Surowiecki has noted, individual judgment aggregated this way performs better than a few experts do.

With that in mind, we are introducing pairwise voting to Spigit’s innovation management platform. What’s pairwise voting?

Pairwise voting is a way expressing your preference for one of two items presented. This pairwise comparison can be completed multiple times in cases of multiple items. The concept is used in eliciting consumer preferences, and creates a ranking of a list of multiple items.

Stealing a page from the realm of consumer preferences research, Spigit is applying the pairwise comparison methodology to surfacing top ideas.

How Pairwise Surfaces the Top Ideas

With each presentation of a pair of ideas, a participant is asked to sit in judgment. Specifically, she is asked to judge which idea better meets the objectives of a challenge. Here is the magical thing…she decides which idea better aids the challenge problem based on her own criteria.

That’s right. Participants are able to apply their own cognitive framework to determining the better idea.

This gets to the heart of tapping the wisdom of the crowd. Each person brings their knowledge, experiences and analysis to the job of determining the top ideas. The characteristics that make pairwise effective for surfacing the top ideas:

  • Visibility on current crowd sentiment is muted
  • Greater coverage of feedback on ideas
  • Depth of belief in each idea is conveyed
Let’s examine each of those elements in greater depth.

Visibility on current crowd sentiment is muted

As Surowiecki explains, the best wisdom of the crowd emerges when participants provide their unvarnished view, uninfluenced by others. This honest assessment is key for leveraging the diverse views of participants.

Pairwise offers an effective means of shielding existing crowd sentiment for each idea. Rather than see all the votes cast for each idea, participants are presented with two ideas and a simple choice: which one is better? This isolation of the voting mechanism from the votes tally reduces the influence of crowd sentiment on each person’s choices.

Greater coverage of feedback on ideas

In a website list of items – any items – those listed beyond the first page or too deeply down the first page can suffer from fewer views. This has a consequence of reducing the level of attention and feedback they receive. Search engine results are a well known example of this dynamic.

With pairwise, every idea gets a viewing. The repeated presentation of pairs of ideas means each idea gets a fair hearing, regardless of where it would have been listed among ideas. This improves the percentage of ideas that have material feedback on them.

Depth of belief in each idea is conveyed

With regular voting on ideas, an individual can grant up votes to multiple ideas. This is a low friction, easy way to express idea preferences, But since each vote counts the same, there is no distinguishing which of the up voted ideas that person feels most strongly about.

With pairwise comparison, each decision made between two ideas is an expression of preference. Assume there are two ideas that an individual likes. Each idea does well when put head-to-head with other ideas. When the two liked ideas are presented to the individual, she must decide which one better addresses the challenge.

Repeat this process for multiple liked ideas, by many different participants, and you end up with a ranked order community preference for all ideas.

Pairwise Pro Tips

In our own usage of the pairwise comparison mechanism, we’ve developed good experience with its dynamics. From that experience, we can share tips and describe product features developed in concert with that experience.

Number of pairwise combinations

Each pairing of ideas will be presented randomly to each participant. A good question to ask is: how many combinations will there be?

There’s a simple mathematical equation to determine the number of pairs. Assume you have N number of ideas. The equation is:
Number of pairs = [N x (N-1)] / 2

For example, if there were 10 ideas, there would be (10 x 9) / 2 = 45 combinations.

Managing high idea volumes for pairwise comparison

The pace of presented idea pairs is rapid. Because of this, we have found that you can process a large number of pairs quickly. Still, there may be cases of high volumes of ideas, where even the most determined participant could work diligently without reaching the end of the presented pairs. There are two approaches to managing this.

First, the distributed pairwise voting across all participants ensures coverage for all ideas, even if no individual pairwise votes on 100% of pairs. This occurs because the presentation of pairs is randomized, meaning each participant gets a different set of ideas on which to vote.

Second, the number of ideas subject to pairwise comparison can be reduced within a Spigit community. Specifically, pairwise can be limited to ideas in a specific lifecycle stage. For example in Stage 1, there may be 250 ideas. After the crowd helps identify the best of these ideas through traditional up/down voting, 12 ideas may move through to Stage 2. At this point, only the 12 ideas can be subject to pairwise comparison.

Three use cases for pairwise comparison

Viewing the experience and application of pairwise comparison, three use cases are apparent

Community’s all in on all ideas: The entire community pairwise votes for all ideas, right in the opening stage. Provides a fresh alternative to the usual voting mechanisms. The top ideas emerge through pairwise comparisons, similar to the way consumer preferences emerge in market research.

Community pairwise compares second stage ideas: After a vigorous round of up/down voting and commenting, the top ideas graduate forward to the second stage. In the first stage, the participants freely gave up votes without regard to rank ordering the ideas. In the second stage, they have an opportunity to express preferences among the ideas. It’s a second round of fresh voting, albeit via a different means.

Selected roles pairwise compare later stage ideas: The community does the initial work identifying ideas with the highest potential. These ideas move from the first stage to the second stage or beyond. At that point, a select group of individuals (e.g. experts, senior executives, etc.) pairwise compare the ideas. This use case takes on aspects of portfolio management in prioritizing the ideas.

To learn more about pairwise comparison, contact us. Or, try out the pairwise experience for free with Spigit Icon.

I’m @bhc3 on Twitter. Tagged: customer centricity, customer insights, customers, innovation, voc, voice of the customer