Crowdsourced Opinions vs Expert Knowledge

December 15, 2014

By: Holly Stark

Planning a recent vacation led me to a multitude of online reviews. I combed through Trip Advisor for hotel appraisals, Chowhound, Eater, and Yelp for restaurant reviews, and local city-centric sites for cultural offerings. Crowdsourced opinions are a boon to travel planning, but are they accurate? Crowdsourced investment research is taking hold on Wall Street too, as investors look for alpha-generating information outside traditional research generated by investment banks. But the question for both: Does crowdsourced data lead to superior outcomes, be it vacation experience or investment performance? Is relying on the collective intelligence of an online community the key to making an informed decision?

Not necessarily so according to a paper in Science, entitled ‘Social Influence Bias: A Randomized Experiment’. Authors Lev Muchnik, Sinan Aral and Sean J. Taylor looked at posts on a social news aggregation website and found that “Prior ratings created significant bias in individual rating behavior, and positive and negative social influences created asymmetric herding effects.” If a poster made positive comments, the next poster was likely to make the same. If the same holds true for hotel and restaurant reviews, would the ratings be skewed by prior posts?

Investors too are increasingly relying upon aggregated reviews and the collective intelligence of investment analysts and online commentators to ferret out investment ideas and superior performance. Crowdsourced research platforms using various strategies to harness data are gaining attention as studies show that they can generate average-beating returns – but evidence suggests that participants may be ‘talking their book’ to attract more capital into a particular stock.

Pre crowdsourced reviews, a savvy, plugged in travel agent would be the best source of travel info provided the agent’s firsthand knowledge was current. A Wall Street equivalent to an experienced travel agent, ‘Expert Networks’ are people who have specific knowledge about a company or industry – though the networks have come under fire from illegal disclosures of non-public, inside information. But in his book Expert Political Judgment: How Good Is It? How Can We Know? Philip Tetlock shows that predictions by recognized experts performed only slightly better than the average person who is more casually informed about a subject.

Our research identifies people who are knowledgeable and engaged in a company and the industry it operates in, but they aren’t necessarily the publicly recognized expert. We’ve found that these distinct people are predictive of what their peers will do – very useful information to harness when making investment decisions. We ask predictors to measure Competitive Momentum: if a company is gaining ground, losing ground or staying the same versus its competitors. Our work shows this measure is an indicator of reputation and has a high correlation with sales revenue, market-share changes, stock movement and GDP trends, providing better, more actionable information.

The performance results of my scrutiny of crowdsourced travel reviews? Fair. The best recommendations were from friends with similar interests and tastes, current with the latest about a destination. Relying upon their recommendations resulted in superior experiences. It applies to investing as well: people who are deeply immersed in a company – but not necessarily the acknowledged industry gurus – offer a predictive view that is missing from the crowdsourced one.

Holly Stark is Chief Operating Officer of Lepere Analytics. You can reach her at