Harnessing the Wisdom of the Crowd Humans are sociable creatures who operate in crowds. Recent research has helped illuminate the traits and tendencies that separate good crowds from bad.

“Always remember that the same crowd that applauds your coronation is the same crowd that will applaud your beheading. People like a show.” Fantasy novelist Terry Pratchett, Going Postal, 2004

This is a story about crowds that begins in ancient Greece and continues today. The city of Athens, named for Athena, the goddess of wisdom, is considered the world’s first true democracy, with thousands of Athenian citizens (men of a certain age) actively participating in the city-state’s decision-making. Athens built an empire throughout the eastern Mediterranean in part because of the democratic inclusion of its citizens in governance. This meant that in many cases the best ideas could be implemented in policy through the collective wisdom of the crowd. And it ensured that a committed mass of citizenry had a deep investment in the state’s interests, particularly in matters of war.

That, at least, was the theory. The reality was more complex. This same informed Athenian population also charged, convicted and executed the philosopher Socrates and exiled the general Thucydides after he was defeated in battle. In fact, the Athenian assembly, consisting of all of the city-state’s male citizens, drove out almost every gifted general at one time or another. During this period, Athens found itself embroiled in a series of wars against a coalition led by Sparta; eventually, this led to Athens’ defeat and the loss of much of its empire. The greatest analysis of Athens’ ruinous military campaigns was written by that former general Thucydides in his History of the Peloponnesian War. Thucydides, not surprisingly, was no fan of Athenian democracy.

Still, Thucydides and other contemporary observers, including Aristotle, argued that the Athenians’ singular democratic culture ensured resilience from catastrophes, even those caused by the assembly’s decisions. That paradox captures the nature of democracy — and the broader subject of what is known today as the wisdom of crowds. Collective wisdom is the body of knowledge and principles that develops within a society and may be unleashed by democracy. As President John F. Kennedy once said, “For, in a democracy, every citizen, regardless of his interest in politics, ‘holds office’; every one of us is in a position of responsibility; and, in the final analysis, the kind of government we get depends upon how we fulfill those responsibilities.”

Today we recognize the power of crowds in politics and in fields like finance and economics. When we talk about the ability of markets to efficiently process information, to serve as a guide to future prices or to form destructive bubbles, we are talking about the paradoxes of crowd power. Over the past few decades, the study of crowds has emerged as a fascinating and relevant research subject. This article will explore what we’ve learned about crowds and their behavior in different contexts, what constitutes their successes and failures, and what drives them.   

What, exactly, is a crowd? A crowd is a group of people who gather together for some purpose. A crowd may be defined through that common purpose or set of emotions, and it may include partisans at a political rally, fans at a sporting event or the audience at an opera. Crowds don’t necessarily have to be gathered physically together, like the Athenian assembly on the hill of Pnyx. Modern mass communication and social media attract virtual crowds of many kinds, and markets are no longer confined to physical exchanges. There are many different kinds of crowds: an aggregation, audience, group, mob, populace, public, rabble or throng. Each of these has a slightly different purpose and characteristics. Generally speaking, a crowd arrives at decisions regarding action — in what way it will be taken and how quickly — even if only a few of its members have the necessary information about the goal or purpose of those actions. 

A crowd consists of individuals. Crowds can be characterized by how effectively they can make judgments and decisions. A “good” crowd is able to make fair, unbiased, rational decisions, even in cases where there is a deficiency of information. (Decisions may be unbiased and rational without necessarily being correct.) A “bad” crowd may lack some important aspect, such as diversity or independence, and may produce wrong, even disastrous, judgments; a bad crowd may demand conformity. An “ugly” crowd reacts unpredictably, with, probability-wise, a fat left tail.

Today, in our data-driven, digital world, and despite large physical distances, everyone on Earth is connected to everyone else by up to six degrees of separation.1 The growing density of human networks makes actual social distances much smaller. The degree to which informed members can affect a crowd depends on their position within the group, with those in the crowd’s core likely to have a greater influence. An interconnected world generates enormous amounts of data. Increasingly, people aim to make use of this data to improve the quality of life, and aggregation is one way to do that. But what data is actually useful? How do we effectively aggregate data? These are fundamental questions that the study of crowds may well help to illuminate. 

Good and Bad Crowds

“The intelligence of that creature known as a crowd is the square root of the number of people in it.” — Terry Pratchett, Jingo, 1997

The study of crowds is a subject within the social sciences. Scott Page’s book The Difference: How the Power of Diversity Creates Better Groups, Firm, Schools, and Societies makes the case that the effectiveness of crowds depends on diversity — not what we appear to be from the outside but what we look like from within (that is, our individual abilities and tools).2 Page, a University of Michigan professor of complex systems, political science and economics, argues that progress and innovation may depend less on lone thinkers with high IQs than on a diverse group of people working together and capitalizing on their individuality.

James Surowiecki, author of The Wisdom of Crowds, agrees with Page on the importance of diversity. Decisions resulting from the aggregation of information in groups, he writes, are often better than those made by individuals.3 Surowiecki, a financial writer for the New Yorker, focuses on both diverse groups making independent decisions and manifestations of crowd psychology, such as herding, in which clusters of individuals follow the larger crowd. The argument for “good” crowds is that a diverse collection of individuals deciding independently is likely to make certain types of judgments and predictions more effectively than individuals — even individual experts — alone. This process has many parallels with statistical sampling, in which you try to understand the general tendencies of a larger population by choosing a smaller, representative sample.

Surowiecki posits that five elements are required to form a wise crowd: diversity of opinion, independence, decentralization, aggregation and trust. Having a diverse group of decision-makers, he writes, “helps because it actually adds perspectives that would otherwise be absent and because it takes away, or at least weakens, some of the destructive characteristics of group decision making.”

Diversity refers to individuals in the crowd having access to private information, even if it’s just an individual interpretation of commonly known or accepted facts. To be independent is to be free of undue influence by others in the use of that information. “Collective decisions,” writes Surowiecki, “are most likely to be good ones when they’re made by people with diverse opinions reaching independent conclusions, relying primarily on their private information.” Surowiecki defines decentralization as a situation in which “power does not fully reside in one central location, and many of the important decisions are made by individuals based on their own local and specific knowledge rather than by an omniscient or farseeing planner.” Decentralization allows individuals to specialize and draw on local knowledge. Aggregation is the mechanism that turns individual judgments into a collective decision. The collective group operates best when it’s fair and capable of encouraging broad-based trust among decentralized strangers.

Research routinely attributes the superiority of crowd averages over individual judgments to the elimination of the kind of noise associated with cognitive biases — an explanation that assumes the independence of individual judgments. Thus, a crowd tends to make its best decisions if it is made up of diverse opinions of individuals operating independently. Page formulates a kind of diversity prediction theorem: “The squared error of the collective prediction equals the average squared error minus the predictive diversity.” Simply put, that means the greater the diversity, the lower the collective prediction error.

Averaging can eliminate random errors that affect an individual’s answer, but not systematic errors that affect the opinion of the entire crowd. Therefore, an averaging technique could not be expected to compensate for cognitive biases that drive a crowd to act or react in a certain way.

Practical Applications

Research into the study of crowds has applications in three broad categories. The most common application is the so-called prediction market: speculative or betting markets created to make verifiable predictions. For example, Dublin’s Paddy Power Betfair, a unit of bookmaker Flutter Entertainment, is the world’s biggest prediction exchange. Prediction (or information) markets ask questions like, “Who do you think will win the election?” and they predict outcomes rather well. Individual answers to the question “Who will you vote for?” are not as predictive. Prediction markets interpret current market prices as the probability of an event occurring or the expected value of a parameter. Assets are cash values tied to specific outcomes — for example, that a certain candidate will win the election — or parameters, such as next quarter’s revenue.

The second application is the so-called Delphi method, developed by Rand Corp. in the early 1950s. This is a systematic, interactive forecasting method that relies on a panel of independent experts. Carefully selected experts answer questionnaires in two or more rounds. After each round, a facilitator provides an anonymous summary of the forecasts and reasoning from the previous round. Participants are encouraged to revise their earlier answers in light of the views of other members of the group. During this process, the range of the answers tends to narrow and the group converges toward a “correct” answer. Many of these consensus forecasts have proved to be more accurate than predictions made by individuals.

Third, human swarming is an optimized method for unleashing the wisdom of crowds. This approach implements real-time feedback loops around synchronous groups of users, with the goal of achieving more-accurate insights from lower numbers of people. Human swarming is modeled after biological processes in birds, fish and insects, with a network of users employing a mediating software, such as the UNU collective intelligence platform. Such real-time control systems allow groups to behave as a unified collective intelligence. In fact, we may think of financial markets in much the same way, as much a part of nature as beehives or ant colonies, in which swarm intelligence arises despite individual agents not fully understanding the rules distributed among them all.4

When does collective intelligence fail? Centralization can lead to biased results and cripple collective intelligence when decision-making is not open to individual judgments and the benefit of aggregating unique private information is lost. Another source of failure: errors in sampling, in which you sample from a different distribution from the one that really exists; this can produce collective hysteria. And when the crowd is homogenous, its intelligence may be biased. Diversity helps ensure there is variance in approach, thought processes and private information.

The most striking example of a crowd failure is the rational bubble. In this situation, collective cognition and cooperation fail because, in one way or another, members of the crowd grow too conscious of the opinions of others and begin to emulate one another and to conform rather than think independently. John Maynard Keynes identified this as an aspect of markets in his metaphor of investors as beauty pageant judges who try not to pick the most beautiful contestant but the one the other judges would choose. When independence declines, imitation and herding kick in and diversity vanishes.

While there is merit to the idea that the many are smarter than the few, it is not always true, particularly when members of a crowd are aware of and overtly influenced by one another’s opinions. Consensus thinking among a group with poor judgment can lead to disastrous group decision-making. This factor may have been among the causes of crises ranging from the 2008 financial meltdown to Athens’ defeat in the Peloponnesian wars.

Markets as Crowds

In a 2015 column, wealth manager and columnist Barry Ritholtz argued that, unlike markets for equities and goods and services, prediction and futures markets lack the wisdom of crowds because they do not possess a large or diverse enough pool of participants.5 He pointed out that prediction markets failed spectacularly in trying to guess the outcomes of events such as Morgan Stanley CEO Philip Purcell’s 2005 resignation and the 2015 Greek referendum on a European Union bailout. Individuals trying to predict the outcomes of these events were simply guessing, based on public polling data, and lacked any special individual or collective knowledge.

Research into crowds also explains what makes markets efficient at some times and inefficient at others. Strong efficiency is a chimera, but the efficient market hypothesis, which states that prices incorporate all available information on an asset price, is somewhat true in its weaker form; that is, all existing information is not yet assimilated but will be assimilated by the market. That information, in turn, will promptly remove imbalances — a process known as statistical arbitrage — and rationalize a security’s or product’s price.

When this process breaks down, markets will be less efficient and prices will separate themselves from value. That’s often the case with rational bubbles; peer pressure, imitation and herding vaporize diversity and independence. Markets may crash after an asset bubble bursts, when investors fully realize the unsustainability of market prices. That’s when panic sets in and market liquidity evaporates.

Portfolio diversification is the most powerful tool to reduce risk, but it is simply not enough to protect investors from a systemic failure.6 When investors start pulling money out of highly leveraged hedge funds, the latter are forced to liquidate large portions of their portfolios at the same time. The withdrawal of liquidity and the unwinding of leverage hit all asset classes except government bonds and cash, which can provide real diversity.

Yet, for much of the time, markets may appear, through active and rational arbitrage, to be relatively efficient. As proponents of the wisdom of crowds argue, a sizable amount of individual irrationality is canceled out in the totality of judgments made in markets. Although markets may not be fully efficient, they are a kind of continuous voting mechanism that passes a stream of verdicts on consensus value.

Conclusion

Crowd behavior is a fascinating phenomenon, which is omnipresent in our world despite its elusive qualities. From ancient to modern times, consciously or not, we have judged, negotiated, operated and lived in crowds. Humans are sociable creatures, and crowds are the waters that we swim in. We gather in groups where we communicate, analyze and make choices together, from fashion trends to lifestyle choices to political leaders. We go to war in crowds, and we make peace in crowds. Our economy functions through interlocking crowds — markets. The crowd shapes every corner of human life, from a range of social activities, including democratic politics, to financial markets and even to advanced techniques like machine learning.

But crowds, like the individuals that make them up, are imperfect. When certain conditions in the crowd ecosystem are met — diversity, independence and the ability to tap private information — crowds can act with intelligence, even wisdom. When these conditions are limited or eliminated, crowds can become foolish, blind, even dangerous. In a homogenous crowd, cognitive biases can undermine the predictive power of crowdsourcing. A crowd that has been seized by emotion, that has lost its ability to think and judge, may turn lethal.

The crowd is constantly evolving, just like humanity itself. Given our human nature, our individuality married to a need to engage socially, we really have very little choice but to operate in crowds. The way may be difficult, but harnessing the wisdom of the crowd may be the only path to a happy, prosperous and successful society.

 

Michael Kozlov is Senior Executive Research Director at WorldQuant, LLC, and has a Ph.D. in theoretical particle physics from Tel Aviv University.

Radoslav Valkov is a Senior Quantitative Researcher and has a Ph.D. in mathematics from the University of Antwerp.

 

ENDNOTES

1. Frigyes Karinthy. Láncszemek (Chain-Links). 1929. 

2. Scott E. Page. The Difference: How the Power of Diversity Creates Better Groups, Firm, Schools, and Societies. Princeton, NJ: Princeton University Press, 2007.

3. James Surowiecki. The Wisdom of Crowds. New York: Doubleday, 2004.

4. Igor Tulchinsky. The UnRules: Man, Machines and the Quest to Master Markets. Hoboken, NJ: John Wiley & Sons, 2018.

5. Barry Ritholtz. “The ‘Wisdom of Crowds’ Is Not That Wise.” Bloomberg (July 7, 2015). 

6. John Plender. “The Flawed Wisdom of Crowds.” Financial Times (June 16, 2009). 

 

Thought Leadership articles are prepared by and are the property of WorldQuant, LLC, and are being made available for informational and educational purposes only. This article is not intended to relate to any specific investment strategy or product, nor does this article constitute investment advice or convey an offer to sell, or the solicitation of an offer to buy, any securities or other financial products. In addition, the information contained in any article is not intended to provide, and should not be relied upon for, investment, accounting, legal or tax advice. WorldQuant makes no warranties or representations, express or implied, regarding the accuracy or adequacy of any information, and you accept all risks in relying on such information. The views expressed herein are solely those of WorldQuant as of the date of this article and are subject to change without notice. No assurances can be given that any aims, assumptions, expectations and/or goals described in this article will be realized or that the activities described in the article did or will continue at all or in the same manner as they were conducted during the period covered by this article. WorldQuant does not undertake to advise you of any changes in the views expressed herein. WorldQuant and its affiliates are involved in a wide range of securities trading and investment activities, and may have a significant financial interest in one or more securities or financial products discussed in the articles.