It was hardly news to long-time followers of Robert Hodgson’s painstaking science, but the winers’ shock-horror nyah-nyah I-told-you-so flash-in-the-attachments spitbucket has again revealed some of his findings relative to wine judges.
Like, fair dinkum, wine judging is not science.
Hodgson is a retired oceanographer with a background in statistics and small winery in California. His inquiring mind has led him to maintain a study of the vagaries of wine show results. Commencing with an analysis of the range of scores his own Fieldbrook Winery products won in various competitions, he went on to mount larger-scale testing of the judging results over entire wine shows.
Hodgson’s eight years of methodical analysis reveals that even the most respected wine judges vary widely in their scoring of wines, not just from week-to-week and day-to-day, but from minute-to-minute.
“The results are disturbing,” Hodgson told David Darbyshire of The Observer. “Only about 10% of judges are consistent and those judges who were consistent one year were ordinary the next year.”
Unsurprisingly, this variation increases significantly as the number of wines judged in each sitting multiplies, which is why most wine shows are judged by panels of judges working together. Typically, their scores for each wine are averaged to achieve a final ranking. They keep a check on each other.
My most recent experience in this arcane world of wine races occurred last week when I assisted in the selection of McLaren Vale wines to be poured at the next Gorgeous Festival. The wines were all served blind. After a large mixed class of what were loosely called Reds – Anything But Shiraz, I was disappointed to the point of depression. When we six judges retreated to discuss our findings, this feeling was common. Regardless of the wide range of “new” and “alternative” grape varieties on display, and the stalwarts like Cabernet sauvignon and Grenache, it had been very difficult to find anything of exceptional quality, and we all shared the feeling that we doubted our palates, and confused and exhausted ourselves re-examining and scouring the lines of glasses in search of something outstanding which we all believed should be there but wasn’t.
Admittedly, many of my personal favourites from the region were not entered: some winemakers didn’t have the volumes required, preferred to keep their stock for sales to regular buyers, like at cellar-door, or simply forgot to enter. Nevertheless, there were dozens of wines there from reputed large producers, and many from the smaller wildcats, cult heroes and respected pioneers.
If the same selection had been poured to the judges at the annual McLaren Vale Wine Show, half the entries would have been awarded medals. While these would be mainly bronze, simply indicating the wine had no overt faults and was technically sound, there would (on average) be several silvers, a gold or two, and a trophy.
If our team of tasters was composed of local winemakers rather than experts who don’t make wine, I have no doubt the results would have been more sympathetic to the competitors.
Whoreses for causes, see?
My mantra for the selection of wine show judges is simple. Any winemaker who doesn’t recognise their own wines in a blind tasting should not be judging – their organoleptic skills are inadequate. On the other hand, any winemaking judge who fails to award their own wines well, or wines of similar styles, should not be expecting us to buy them.
There’s been an increasing attempt around the burgeoning wine-racing circuit to select expert judges who are not winemakers, or at least to import winemaking judges from other regions, but it’s still too easy to influence the results by inviting judges who the selection panel knows to be sympathetic to certain styles of wine.
Then, of course, as Hodgson’s work reveals, you risk the variable induced by organoleptic exhaustion in the bigger classes, a problem which can be partly diminished by selecting bigger tasting teams or better ones, but even this ends up levelling the results because the scores are averaged. Derbyshire also reports the works of French academic Frédéric Brochet and American economist Robin Goldstein, who independently proved wine judges award higher scores to wines they perceive to be more expensive. And he mentions British psychologist Professor Richard Wiseman, whose testing showed that of 578 people invited “to comment on a range of red and white wines, varying from £3.49 for a claret to £30 for champagne, and tasted blind … People could tell the difference between wines under £5 and those above £10 only 53% of the time for whites and only 47% of the time for reds”, meaning that “overall they would have been just as a successful flipping a coin to guess”.
My research is slightly different. I was once invited to present a tasting to a large conference of hardrock mine managers and mining industry executives and directors from all over the world. We did this over a four-course dinner. I selected two good wines to harmoniously accompany each course; one under $10 and one over $30. Nobody knew their type or identity. In the vast majority of cases, these red-loving, high-earning men thought the cheaper wine to be the over-$30 job.
Knowing the parsimony, if not the sheer meanness, of mining executives, and I’m only half-joking, this indicated a presumption that wine of the price they commonly drank seemed the best, and therefore most valuable, to them. This test was hardly scientific in the clinical sense, but it gave me a fairly good idea of how many enthusiasts simply get things wrong. While all the wines were Australian, there was no discernable difference between the opinions of the Australian drinkers and those of the internationals.
All this led me long ago to withdraw from the giant wine-racing circuit, and sit at home with my glass and the endless string of bottles that arrive unsolicited, and simply get on with explaining which ones I prefer and why. And yes, without tasting blind, I award them scores. It’s a one-man continuous field wine race, and it’s a long course. Unlike the quick glimpse of each entry the judges have time to take in the big national sprints, at least I get to savour each wine over many days, to best grasp their nature and potential. In fact, I watch them run until they die.
So trust me if you dare.
The Gorgeous Festival, by the way, will take place beneath the red gums at Serafino Winery on Friday and Saturday, November 22-23. I can guarantee the wines served there will all be of superior quality. If you doubt me, enjoy your water, and see how you go scoring the music. To my palate, the organisers are lining up some damn fine acts.
Keep an eye on the website.
