Why monkeys throwing darts are as good as wine experts
We all buy Gold Medal winning wines, but Dave Robbins discovers you may as well pour food colouring into an old bottle of Chardonnay
Published 15/07/2014 | 02:30
It turns out the chance of winning gold is the same as from a random process
You go into the off-licence in search of a decent bottle of wine. Perhaps you're having people over, or going to a dinner party. You want to impress, to give the impression that you know something about wine. The problem is: you don't.
So you look at the "wall" of industrially produced wine. The bottles and labels begin to blur into one. Then a sticker catches your eye: "Gold Medal Winner". You buy it. It's got to be good, right?
Wrong, according to statistician-turned-wine grower Robert Hodgson. Hodgson, now possibly the most hated man in the Californian wine business, has proved that the chances of an award-winning wine being any good are no better than random chance.
His research, carried out over four years at the prestigious California State Wine Fair, has shaken the wine world and begs the question: is everything we know about wine wrong?
So what did he do? Well, with the permission of the organisers, he slipped three identical wines to each judge without them knowing. Would they be able to spot the three that were the same. Eh, no.
"We gave each judge a flight of 30 wines," says Hodgson. "They'd all have three samples of the same wine, but they didn't know it. These samples were arranged randomly. When I first saw the results I could hardly believe them. They scored the identical wines like they were different. It was staggering."
On a 20-point scale, an identical wine would typically vary by four points from one tasting to the next. "About 10 per cent of the judges were really bad," he says. Their judgments of these wines ranged between 16-18 points. "But about 10 per cent were quite good. We thought we'd be able to use these judges as mentors to teach the others how they did it."
It didn't work out quite like that. "It turned out they couldn't maintain that performance. One year they might be really good, the next they were just in the middle of the group."
Hodgson broadened his investigation. "I looked at a set of data that showed the scores for wines that were entered into as many as 13 different competitions," he says. "I tracked the scores from one competition to another. There were, like, 4,000 wines that I looked at. Of all the ones that got a gold medal, virtually all got a 'no award' some place else. It turns out that the probability of getting a gold medal matches almost exactly what you'd expect from a completely random process."
Hodgson, a retired academic, has amassed a credible data set. But he's not the first academic to have come to a conclusion that embarrasses the wine community. In 2001, Frederic Brochet put some red food dye in white wine and served it to 54 "experts". None of them spotted the deception.
So is the "gold medal" sticker really worthless? Caro Feely, who left Ireland to make wine near Bordeaux with husband Seán, thinks so. "It's totally random, who wins," she says in a phone interview from her winery at Haut-Garrigue. "It's like a monkey throwing darts at a dartboard – once in a while it's bound to hit the bullseye."
Feely believes wine competitions are weighted in favour of the big, commercial producers. "Small producers can't compete. They can't enter, because there is a minimum threshold for production. In the Bordeaux Wine Competition, I think it's 5,000 litres of a single wine. Then they sell the stickers for about ten cent, and they have to be able to sell a certain amount of them."
She and her husband have won medals in smaller competitions, ones aimed at the smaller organic and biodynamic wine market. Feely, whose second book about her experiences as a vigneron (Saving Our Skins, Summersdale £8.99) is published next month, doesn't even bother to put the stickers on her bottles.
Caro Feely also points out that, at competitions, wine is tasted without food. This means that "the kinds of wines that win are fruity, sweet and alcoholic. Now, you can drink a glass of that on its own, but you probably couldn't drink several glasses over the course of a meal.
Evelyn Jones, chairperson of the National Off-Licence Association and owner of The Vintry off-licence in Dublin 6, is not so damning of wine competitions.
"People do like to see the medal stickers on wines. It helps people to navigate through the world of wine. People are time-poor. They want to enjoy a really good glass of wine without having to sift through mediocre wines to find one," she says.
"Look, when you get into the top echelons of the wine market, each wine is unique. But in the middle ground, there is a huge amount of wine produced and these wine competitions help the consumer by sifting through these mediocre wines and picking out the best.
"And the marks are averaged out, they're consensus scores, so I would say that the competitions and awards are a useful shortcut for time-poor consumers," she says.
The issue of wine tasting being "junk science" has been widely considered in the wine blogosphere. The controversy began, of course, with the publication of Hodgson's research, which was followed up widely in the US media.
Respected wine judge and blogger Robert Joseph says the wine community is guilty of presenting its awards in a scientific way – by, say giving a wine 79 instead of 80 points, which "implies considerable precision," he says.
He also points out that "identical" wines can vary greatly, depending on multiple factors. So perhaps wine tasters should admit these variables up front and point out that wine tasting can't be "junk science" because it's not science in the first place.
As for Robert Hodgson, he now makes wine at his Fieldbrook Winery in Humbolt County, California. And yes, he enters his wines into competitions. "I still feel good when I win a medal. I do! I crow about it like everyone else. When I lose, I don't get upset about it. Not any more."
DEARER DOESN'T MEAN BETTER
Robert Hodgson's interest in wine began when he and a group of friends began to pool their spare cash to buy wines to taste.
"I got a bunch of people together and we would all chip in and buy expensive wines to see what they tasted like.
"We had – and I'm not kidding here – a Château Latour, as well as a good Californian and a real cheap one.
"We found that $5 (€3.67) wine is not very good, $10 (€7.34) gets better, $15 (€11.02) is pretty damn good but if you go up to $20 (€14.69) it doesn't get any better from there to $200 (€146.98)."
THE POWER PALATES
Decanter Magazine and the wine and drinks industries both publish annual lists of the most influential people in the wine business.
Most are producers, shippers or buyers, people whose influence is commercial. Many are now Asian, reflecting the huge influence the Chinese market is having on the wine world.
But two ever-presents on these lists are "tasters": Robert Parker, the US wine writer, and Michel Roland, the French wine consultant.
Together, these two have a huge influence on the style of wine the world is drinking.
"Parker Points" can make or break a winery, while Roland, the go-to consultant for wineries from Bordeaux to Brazil, is just as significant on the production side.