Brain power

Using prediction markets to verify research

Originally posted on The Horizons Tracker.

Last year I covered a project by researchers at Stanford and Carnegie Mellon to use citizen scientists to verify the results of academic research.

Participants played a relatively simple game that the designers hoped would prove the rigor of various studies being put under the microscope.

Using the crowd mkII

Whilst that project used game based approaches, a recent project undertaken by Harvard is using prediction markets to put research through its paces.

The prediction market was used to estimate the reproducibility of over 40 experiments that had gained prominence in psychology journals.

The crowd proved to be good at predicting such reproducibility, with a 71% success rate on the sample given to them.

“This research shows for the first time that prediction markets can help us estimate the likelihood of whether or not the results of a given experiment are true,” the authors say. “This could save institutions and companies time and millions of dollars in costly replication trials and help identify which experiments are a priority to re-test.”

The results are not quite so positive for the research industry itself, with a worrying 61% of studies failing to reproduce the original results.

A quick and effective way of testing

The authors believe, therefore, that tools such as prediction markets can prove invaluable in offering a quick and easy way of testing findings.

“Top psychology journals seem to focus on publishing surprising results rather than true results,” the authors say. “Surprising results do not always hold up under re-testing. There are different stages at which an hypothesis can be evaluated and given a probability that it is true. The prediction market helps us get at these probabilities.”

Of course, prediction markets are a fairly mature methodology these days, with successful applications in areas such as medicine, economics and even in testing science and technology predictions.

The experiment was part of the The Reproducibility Project: Psychology, which is an open science project designed to test the reproducibility of psychological research.

Participants were given $100 to invest in the market and given information about the studies, including the full paper, info about the authors and so on. With this information, they could then invest on the likelihood of reproducibility.

“One of the advantages of the market is that participants can pick the most attractive investment opportunities” the authors say. “If the price is wrong and I’m confident I have better information than anyone else, I have a strong incentive to correct the price so I can make more money. It’s all about who has the best information.”

The researchers believe that the initial results from the market were positive and offer the potential of scaling things up further to test the approach itself. The authors hope to conduct similar trials in other subject areas to see whether the success is equally applicable in other domains.

“Our research showed that there is some ‘wisdom of the crowd’ among psychology researchers,” the authors conclude. “Prediction accuracy of 70 percent offers an opportunity for the research community to identify areas to focus reproducibility efforts to improve confidence and credibility of all findings.”

Source: Using prediction markets to verify research.

Adi Gaskell

I'm an old school liberal with a love of self organizing systems. I hold a masters degree in IT, specializing in artificial intelligence and enjoy exploring the edge of organizational behavior. I specialize in finding the many great things that are happening in the world, and helping organizations apply these changes to their own environments. I also blog for some of the biggest sites in the industry, including Forbes, Social Business News, Social Media Today and Work.com, whilst also covering the latest trends in the social business world on my own website. I have also delivered talks on the subject for the likes of the NUJ, the Guardian, Stevenage Bioscience and CMI, whilst also appearing on shows such as BBC Radio 5 Live and Calgary Today.

Related Articles

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.

Back to top button