A faux prediction market hits the dust -- Predictify closes

Predictify, a prediction platform that claimed to be "like fantasy sports for everything else" has cited the tough economic climate and announced its decision to close. Capitalizing on the public obsession with news, the site offered a "forward looking dimension" to news reporting -- allowing users to predict events before they happened (!) The site's failure is not surprising.

Predictify asked a wide range of questions. A recent look at the site shows the following questions as among the most popular:

• Will a cure for Alzheimers be discovered by 2015?
• Will the national drinking age be lowered to 18 in the next two years?
• By the end of 2009, what will be Texas' status in the union?

The site also added demographic information to the predictions in an attempt to make the site more fun. I learned, for example, that 25% more Muslims than Mormons expect Hannah Montana to become a billionaire by age 18. I'm not sure what great insight can be gleaned from this observation, even apart from the fact that in a pool of 1,200 total bets, there were surely not enough Muslim and Mormon bettors to make that result statistically significant.

Predictify was one of a whole host of "prediction market" news sites to appear online with the buzz surrounding James Surowieki's The Wisdom of Crowds. They suffered from some common weaknesses:

• Weak incentives -- relying on a leaderboard when you have thousands of anonymous people who don't know or care about each other is not a particularly effective incentive
• No market making mechanism -- unlike in a real market, where if you want to buy something, someone has to be willing to sell it to you, these sites allowed all transactions to occur, making them more akin to a poll than a market
• Stupid questions -- it's partially the fault of the user base who suggested the questions, but there were so many poorly worded serious questions or simply frivolous pop culture questions, that the sites felt more like a cheap, relatively amusing place to waste time than a serious place to think about the probabilities of future events that matter

The greatest fault of all, however, cuts most directly at the legitimate ability of a site like Predictify to generate accurate forecasts: there is zero reason to believe that the site's user base has any insight into the questions at hand. What level of wisdom across pop culture, political, sports, and business events can be expected from the same anonymous crowd?

This is not to say that collective intelligence is not a powerful tool. In Surowieki's book, the famous opening example of a bunch of farmers at a market better able to collectively estimate the weight of a cow than any individual (i.e., the average estimate was nearly exactly right when no single estimate was) is powerful. The difference? These people actually knew something about cows.

By contrast, I have no insight into Jon and Kate's marriage -- there is no reason to think that asking 2,000 people like me whether they will divorce in 2009 would produce a more accurate prediction than say, asking a writer at US Weekly, who follows them around. A crowd in itself does not produce wisdom.

So the novelty of Predictify has worn down and the site is closing. I hope this signals a change in the public consideration of the prediction market industry -- away from silly news forecasting sites with vapid promises of wisdom and towards serious enterprise prediction markets where the market players possess unique information that effectively pooled can produce insights into real company issues.

Comments

[Conrad]

I agree with everything you say, except your statements regarding an uninformed user base.

With the proper incentives (i.e. forcing people to put money on the line) you can find people with valuable informations pretty quickly, even if the base as a whole is uninformed. However, without money on the line sites like predictify lack the necessary incentives.

[Melody]

Conrad,
I agree that groups of people can be surprisingly good predictors of events. My main problem with the site is that they overwhelmingly asked questions that people are not particularly informed about, like celebrity decisions, or that experts are simply much smarter on, like major macroeconomic indicators. As you point out, the incentive structure was also clearly lacking.

[Martin]

"no market making mechanism" is a curious insight.

The flip side of incentives(rewards) is risk... You might say that any system without risk couldn't possibly be a market.

A little off subject... I've been thinking about prediction markets for "project management" where incentives/risk are tied to compensation. Possibly allow employees to buy shares in projects.

It might be a way to both finance and predict success. Anything commercially available that does this?

[Melody]

Martin-
Check out Inkling Markets. They haven't tied predictive power to compensation as far as I know, but they do have other corporate incentives, like lunch with the CEO for top predictors.

[JT MAloney]

Hi - Good post. You concerns are not isolated to Predictify. They span PMs. Also, I would not care to correlate the firm's meltdown to these valid market defects. There are a million other reasons firms fail, and I suspect Predicity fell victim to one or many of those. Cheers, -j http://www.pmcluster.com/

[Melody]

JT - That is spot on. One thing that I initially failed to mention that has since been brought to my attention is that Predictify also burned through $4M in venture capital in 1.5 years. Definitely more went wrong than the market defects I highlighted.

[Branedy]

There is no wisdom in a crowd. The phenomena is really the reconciliation of 'EXPERT' opinion. The 'market' as such and the monetary element involved is only a method to qualify and quantify an individuals belief in their own expertise in a subject area. Anyone can render an opinion, but without personal involvement in the subject which will denote levels of expertise in that subject will that opinion carry value. The more that person believes their position is accurate the more likely they are to 'bet the farm' and rendering multiple opinions of this nature is what creates the 'Crowd' effect. Read Delphi Technique.

[lookinside]

I spent a lot of time on Predictify. My son in law pointed it out to me. He works in Palo Alto and may know one or both of the founders(never asked). Anyway, it was an exciting experience at first. I apparently was luckier than most, or a bit better informed, as I ended up in the top ten money earners on the site($330). Predictify made one serious error. Each question had comments posted about it. This opened the doors for every dittohead in the country to post inanities. Pretty negative stuff, mostly. A small minority were exceptionally vitriolic, posting to nearly every question. I believe this negativism killed the interest in posting "paid for" questions, about eight months before the company's demise. It was an interesting experiment, and I'd welcome doing something similar, in the future.

[Clotario]

As a former user of the site, I am not surprised to see it folded. The most interesting time for Predictify was the 2008 election, after the election was done and the cabinet posts were filled, there was no reason to keep going back. The hard-right Fox-news set was there in force, and reading the comments (and many of the questions) became intolerable.

Alas, the everyday news simply isn't worth thinking about, much less dealing with morons to predict the outcome of.