Posts Tagged ‘Crowdsourcing’

In this second part of this discussion we will talk about how applying a structured decision process can help to temper, if not overcome some of the biases we pointed out in the previous post.

Let’s start with a brief description of a structured approach to decision making.  For this example we will consider the Analytic Hierarchy Process (AHP) developed by Dr. Thomas Saaty.  This approach has been made accessible by Decision Lens, Inc., who has developed a software solution that enables application of this process to organizations to collaboratively make complex multi-criteria, multi-stakeholder decision where these organizations are faced with many possible courses of action that can be taken.

The Analytic Hierarchy Process (AHP) is a theory of measurement that uses the principles of comparative judgment, first to make comparisons of decision criteria with respect to some property and then derives scales of priority measurement from these judgments.  In doing this the AHP can be used to derive measurements for anything and the meaning depends on the judgments. This allows creation of a measurement system with regard to the criteria to derive the scales and establish a prioritization framework.

The application of AHP follows a structured, while sometimes iterative, format.  Before we look at our example, let’s briefly review the essential aspects of the AHP process.

  1. Develop a hierarchy or tree of criteria- in clusters from high level categories at the top level to more specific sub-criteria, down to ratings and measures that will be used to differentiate the strategic value of options.
  2. Comparison of criteria to each other using a pairwise approach to establish their relative priorities for use in assessing options
  3. Rating the options against quantitative or qualitative scales that are derived to describe each criterion, and measure how well options reflect the priorities expressed within them.
  4. Optimize the allocation of resources among the options by way of cost/benefit analysis and performing sensitivity analysis to determine the robustness and drivers of decision outcomes.

So how can this impact many of the biases laid out in our previous discussion?  Let’s see.

We will look at an example of how some of the anchoring introduced by our fictional Committee Chair is impacted by item 1) and 2) above.  Similarly these techniques can help with many of the other issues highlighted in the way the group interacts in our example.

Committee Chair:  So we have a decision to make.  Which of these two product options are we going to pursue?  This is a critical strategic decision for us, and key to our ability to hit next year’s numbers.

Ideally the group of stakeholders to the decision would take a collaborative approach to defining a decision goal, and establishing the criteria that best enable achieving the goal. The weights of these criteria can then be derived before going down the narrow path being presented by the committee chair.  In this case, the group would be asked what the key factors that shape their decision are.  They may ultimately conclude that the criteria of “next year’s numbers” is the primary consideration for their decision, but without first defining the goal of the decision and articulating the issues that should shape the framework for making the decisions they can fall subject to this anchoring.  Among the important criteria may be the short term financial impact, but perhaps also the long term impact.  The question is which is more important and by how much?  By surfacing these counterbalancing criteria, and having the group weigh in on which should be the larger consideration in the decision, some anchoring from a single influential member can be buffered.  The group in our conversation goes on to talk about a variety of topics in a somewhat random, circular  and unstructured way; what the customers prefer, how easy it is to manufacture, and the overall technical feasibility of the product, etc.  Considering these other criteria can even further balance the thinking of the group and create a more holistic view of the decision.  Once a determination is made by the group of stakeholders to the decision as to which of these criteria is most important, the options can then be put to the test for how well they meet these criteria and the objectives imbedded within them.

If we look at the nature of the conversation, the key stakeholders are speaking of a decision structure that might look something like the following:

This all becomes very interesting in the process of making the comparative judgments.  It becomes virtually impossible for the group to get into single criteria based arguments when posed with a decision structure like this.  By virtue of the comparison exercise you can’t say something doesn’t matter, you can only say how much more or less it matters than something else.  There are often cases like the following where an outlier may need to defend their position in an attempt to influence a group, or have to concede that the rest of the group has a different point of view.  The question is, is this outlier Henry Fonda in “12 Angry Men” or Jim Carey in “Yes Man”.  In some cases information comes to light in these discussions that either substantiates or negates the position of the person who opposes the group, but one way or another, this kind of conversation moves the group to greater understanding and alignment by making all of these preferences and biases more explicit and transparent.  Making this quantifiable using AHP, may look something like this when engaging on the question, “with regard to this decision, which of the following pair is more important, and by how much?”

Provided the decision makers have the ability to stay independent in their assessments and not play “follow the leader” (which can be accomplished by anonymously sourcing these judgments), we can see that despite the very strong preference of the Committee Chair to have the financial focused considerations trump other factors that the opinions and preferences of the entire group give it only a slightly larger consideration on average than the Customer Preference for instance.  The arguments of others may run along the lines of “if we give the customer a preferred product, they’ll buy it, and we’ll meet our financial objectives.  Anyway, our financial forecasts tend to be less reliable than our customer research”.  Often the person with the extreme judgment will consider the positions of their colleagues and scale back the strength of their preference. In other cases, they may hold their ground, but have to accept that theirs is not the popular opinion.  In any case, voices are heard and fair process is at work.  This approach assures that at least participants have to hear perspectives that may interrupt biases.

Let’s look at the next level down in the decision.  When talking about financial impact, which is more important the short term or the long term?

Here again, we can see that the diversity of opinion in the group allows for some greater consideration of the short term, but the more extreme preference of a single stakeholder (which may be completely justified given their incentives, or political pressures) is again dampened by those who feel the long term should be given a more equal consideration, or even preferential consideration.  Collectively the group leans Short Term, but can create a framework that brings a considerably more balanced perspective to the decision.

Using a similar approach the decision makers, or other subject matter experts that they appoint, can be tasked with evaluating how well each option meets each of these criteria for selection.  The same principles of bringing multiple perspectives together and having a structured conversation about outliers can be used to temper the effect of biases and have a counterbalancing effect on more extreme positions while allowing for all of these perspectives to be considered.

This ultimately has the effect of creating a tangible view of the mental model that a group may be bringing to a decision.  These differences in perspective and strengths of preferences can remain invisible in discussion except for the way they are expressed in disagreement, power driven arguments, and advocacy based discussion.

The basic principles as outlined by James Surowiecki in “The Wisdom of Crowds” can be made real using such an approach.  The use of AHP as a decision framework allows for;

  • Independence:  Each member can independently provide their priorities
  • Diversity of Opinion: The AHP actually promotes the expression of diverse opinion
  • Local Knowledge – Stakeholders from different subject matter expertise are able to evaluate the information through their own lens of local knowledge.
  • Aggregation – The mathematical underpinnings of AHP create a unique aggregating mechanism where the decision makers derive the measurement system and establish priorities in a process of transforming implicit considerations into an explicit value model for their decision.

One possible approach to limiting the influence of biases may be to tackle them head on by bringing them to light with a degree of structure and process rather than allowing them to run wild in an unstructured, circular, power driven, advocacy based debate.

Read Full Post »

This past week on The Hill’s Hillicon Valley technology blog, the founder of Fark.com Drew Curtis is quoted to have said

The ‘wisdom of the crowds’ is the most ridiculous statement I’ve heard in my life. Crowds are dumb

He goes on to say…

It takes people to move crowds in the right direction, crowds by themselves just stand around and mutter.
Everyone I’ve spoken to about this since reading this disagrees with him, so I guess he is right.  Well… that’s kind of what you would have to conclude if he is correct to some extent, isn’t it?
I think it’s interesting to look at the examples that were provided to justify this position, and without opening a political debate given their focus, it raises some interesting questions and a pretty compelling point regarding the nature of crowdsourcing.
In James Surowiecki’s hugely popular and compelling book “The Wisdom of Crowds” (see book club section of this blog), Surowiecki is very clear about, and qualifies, that not all crowds are wise.  They need certain characteristics to be present to make them such.  He outlines in significant detail in the book these characteristics, and they are:
  • The group must be Diverse in Opinion
  • The members of the group have to have Independence
  • They have to be Decentralized so that they can maintain their independence to exercise local knowledge
  • There has to be an Aggregating mechanism to converge the group’s opinion to a single point of view
I think there may be something missing from the formula which we’ll get to in a moment.  What struck me about these comments was that crowds engaged in many of the kinds of forums referred to in the post should possess the first three qualities almost by their nature as web forums.  The exception seems to come in the “Diverse in Opinion” aspect as like minded people tend to gravitate to certain online cultures and in some extreme cases the most frequent contributors to these message boards attempt to run off all but the most persistent non-conformists.  The homophily principle can play a part in limiting diversity in online communities.  Given the present state of technology, there does seem to be a gap in any true aggregating mechanism that can bring together a view of what a group may be saying.  Word Clouds may try to visualize it, and polls can provide some insight but can often be taken many times over.  I’m bothered by something more fundamental that may be missing in this discussion about these kinds of social network and their ability to be wise crowds, and it is…
  • A willingness and commitment, to try in earnest through the crowd interaction, to make a decision, solve a problem or achieve a goal through compromise for mutual benefit .

I think compromise may be an additional corollary to the four conditions of wise crowds when it comes to some aspects of social media.  Stubborn “my way or the highway” opposition needs to be reserved for very special and very rare cases, and can’t be the norm.  There needs to be some kind of appropriate motivation or incentive to participate in earnest, otherwise the divisive power driven arguments that are aimed at silencing people of differing opinions, will likely result in crowds that are quite irrational.  When the purpose of engaging in a discussion in a social media forum is to score points, intentionally obfuscate another’s position at any cost, be decidedly closed off to anything someone of another affiliation than your own offers, or just plain be mean, the crowd will be cynical and not engaged in trying to achieve a compromise, but rather squared off to win at any cost!   In these cases the inability to extract wisdom from a group might be attributed more to a victory of foolish individuals than a failure of wise crowds.  In most of the examples in “The Wisdom of Crowds”, participants either had a self interest in getting the right answer, or were working as part of a group against a well defined goal.  I’m not at all convinced this can’t be the result of emergent leadership within a group that builds this alignment and solicits input toward a goal.  Designing diversity into a conversation and being open and accepting of what it brings to the process of drawing out the wisdom in the crowd certainly fits the definition of leadership.  If however the suggestion is that there a few wise individuals,  smarter than the rest, that need to edit inputs and tell the “muttering” crowds what to do because they don’t know what’s good for them, then that’s another matter.  The semantics of the post are interesting and can be interpreted a few ways, my initial take was that the comments were a bit condescending.  Curious what others think.  Feel free to mutter, and I’ll try to limit editing.

Read Full Post »

%d bloggers like this: