User ID:
Password:

 
Remember me
Lost password?

THURSDAY THORNS
01/19/2017
The Paris Tasting Results—And The Meaning Thereof: Part 2

By Charles Olken

In yesterday’s blog, I got all hot and bothered at the notion that “winning” the Paris Tasting some forty years ago was being called into question. The man in charge of the Recount has now sent me a long and fairly involved explanation of his efforts—and it turns out that he was not actually challenging the grand conclusions but the accuracy of the reported results. Or to be more precise, the accuracy of the conclusion that winning was the purpose and the intended result by looking at the judges’ raw scores for each wine.

His argument, however academic it may be, is more accurate than not in so far as the result of that tasting or any other tasting depends highly on the methodology employed, the adherence to that methodology on the part of the judges and the accuracy of the reporting.

First, his letter to me immediately below, and secondly, his conclusions on his efforts to take a closer look at the raw data:

Dear Charlie:

I said that I would send you some text in mid January, about the Judgment of Paris tasting.
Below is the current draft of my notes.

I don't really write about the fact that some of the data are apparently lost forever. My investigations have lead me to the conclusion that the data as published in Connoisseurs’ Guide are an accurate copy of the data as released by Steven Spurrier. herefore, the apparent errors are in his copy. Since we no longer have the original score cards, we can no longer find out what the results really were.

Instead, I have written about the analysis of the data, and why the announcement of a "winner" is not all that it should be.

Cheers,
David

I'm not sure how a wine "wins" a wine tasting, any more than I understand how people win at gymnastics or ice dancing competitions. I have no idea how or why artistic expression is given a score. Can art really be represented by numbers? After all, art is appreciated based on subjective judgments, not quantitative rankings.

This does not seem to stop people regularly ranking wines that have been tasted together, and declaring a "top wine". But this requires an appreciation based on subjective judgment; and so, either wine is not art, or art can be numbered. At my own wine tastings I do not give scores; indeed, I have never learned how to do so, and I probably never will. I am there to learn about wine, not rate it.

Perhaps the best known of these tasting events, at least to Americans, is the so-called Judgment of Paris, which took place in 1976, organized by Steven Spurrier and Patricia Gallagher. Here, a few wines from France were tasted against a selection of wines from California, and the latter acquitted themselves very well. New World wine was henceforth taken seriously.

Given the outcome of this tasting, it is possibly the third most important event in the social and economic history of wine in the USA, after the imposition and repeal of Prohibition. It was certainly made much of by the media during the US Bicentennial, and this has been repeated every 10 years since. Even the Smithsonian Institution got into the act in 2016, as did the San Francisco Museum of Modern Art.

In tastings like this, each evaluator gives each wine a score (out of 20 or 100, say), and the scores are summed for each wine. The wine with the highest total score is declared the "winner", or at least the best wine of the tasting. This is, unfortunately, exactly what happened at the Judgment of Paris.

The basic problem with doing this to rate wines is that no two people use the same quality scale, even when they are allegedly using the same system. This means that you and I might be in serious disagreement even when we both give some wine a score of 15/20; and we may be in total agreement even if I give a score of 12 and you give a score of 15. To me, 15/20 is 75%, which we labeled as a Distinction when I was a university student; even a score of 13 is 65%, which was called a Credit. However, in the US wine world these would be considered pretty poor scores.

So, we cannot combine scores from people unless they are using exactly the same rating scale. An extreme version of this would be me measuring something in centimeters (as would a Swede) and you measuring it in inches (as would an American), and then having us decide on the "real" measurement by averaging the two numbers. Mathematically, we can do it, but the result won't mean very much.

In order to combine my scale (centimeters) and your scale (inches) we have to somehow standardize the two scales first. Once we have done that, the mathematics immediately becomes meaningful. This is an issue for all activities that involve some sort of value judgment, including competitions like ski jumping, gymnastics and ice dancing. How do we combine the value judgments?

This is the basic problem with the Judgment of Paris. It is patently clear from the results that the 11 people concerned (9 French people, plus Spurrier and Gallagher) were not all using the same scale, even though they gave scores out of 20. Therefore, adding up the scores makes no sense in this case. Those people who use a very wide scale will have more influence on the total score than will those with a narrow range of scores; as also will those with a particularly large or small average score. Simply adding up the scores does not equate with "one person, one vote".

This issue is well known in the field of mathematics; and so you may rest assured that the mathematicians have studied this extensively. Sadly, they have come to the conclusion that there is no single solution to this issue ‹ there are lots of solutions, all with their own pros and cons. The basic dilemma is that the numbers people have listed a series of desirable features for combining scores, and then come to the conclusion that no system can meet them all. This is why you will see all sorts of systems being used in different sporting competitions. Indeed, all sorts of systems are used for voting, since this involves mathematically the same issue.

So, what is the practical outcome for the Judgment of Paris? I have looked at many of the suggested procedures, and applied them to the Judgment data (such as we have; but that is another issue altogether). It seems to me that pretty much whichever way you look at it, the Chateau Montelena 1973 Chardonnay is the top-ranked white wine. However, the top-ranked red wine depends very much on which mathematical system you use ‹ any one of four wines could be considered to be top dog on the day, three of them French and one Californian (the Stag's Leap Wine Cellars 1973 Cabernet).

This is an example of the irony of history. One man adds up some numbers, not realizing that it is inappropriate to do so in his case, and announces his conclusion; and some part of the world will never be the same again. Was there ever a battle won based on such a small horseshoe nail?


 

The CGCW Experience - Take the Tour

Meet the New CGCW

For thirty-five years, Connoisseurs’ Guide has been the authoritative voice of the California wine consumer. With readers in all fifty states and twenty foreign countries, the Guide is valued by wine lovers everywhere for its honesty and for it strong adherence to the principles of transparency, unbiased, hard-hitting opinions. Now, it is becoming the California winelover’s most powerful online voice as well. And, our new features provide an unmatched array of advice and information for aficionados of every stripe.

Comments

paris tasting
by Dan
Posted on:1/20/2017 3:45:02 PM

Irregardless of the statistical outcome of that tasting, the significant result was international recognition and receptiveness to a high quality wine making region other than France. Since then California in particular, ( OK, Oregon, Washington, Australia and Chile aren't doing too bad), has regularly met or surpassed the wines of the "Old World". A direct result of that tasting was that the wine world recognized the global potential for producing world class wines.

tasting
by Thomas Kruse
Posted on:1/21/2017 11:15:07 AM

I have used a 20 point system and stuck by it through changing times and even in the face of criticism, reasoned and not. This letter has made me rethgink my approach to what scoring wine means and how it might best be done if at all.

Rating Systems
by Charlie Olken
Posted on:1/21/2017 2:42:06 PM

The 20-point system is as good and bad as any other. As David Morrison has said so elequently, there is no perfect system, nor is there a system not subject to the potential for grade inflation.

But, as nearly as I can tell, we all like systems of one kind or another whether it is for flat screen tv, movies, wine, restaurants. 

Words, by themselves, are not able to express small differences and preferences. And virtaully any description worth its weight as buying guidance will ultimately come with value loaded words that create a hierarchical universe.

It is not the rating system that bugs me but the grade inflation and the failure on the parts of some reviewers to explain through words, as best they can, what the wine is really like.

Case in point. I today reviewed a Petite Sirah that is so tannic that it can only appeal to a limited segment of the Petite Sirah market, But it will appeal to many, and my scoring, which is higher than my personal feelings about the wine, is accompanied by words that will direct the wine to that segment who want those wines and away from the rest of us. Words are inadequate. So are rating systems.

Putting the two together helps, but subjective is always going to be subjective. But just like democracy, it may not be perfect but its the best we have for now.

Rating Systems
by Charlie Olken
Posted on:1/21/2017 2:42:09 PM

The 20-point system is as good and bad as any other. As David Morrison has said so elequently, there is no perfect system, nor is there a system not subject to the potential for grade inflation.

But, as nearly as I can tell, we all like systems of one kind or another whether it is for flat screen tv, movies, wine, restaurants. 

Words, by themselves, are not able to express small differences and preferences. And virtaully any description worth its weight as buying guidance will ultimately come with value loaded words that create a hierarchical universe.

It is not the rating system that bugs me but the grade inflation and the failure on the parts of some reviewers to explain through words, as best they can, what the wine is really like.

Case in point. I today reviewed a Petite Sirah that is so tannic that it can only appeal to a limited segment of the Petite Sirah market, But it will appeal to many, and my scoring, which is higher than my personal feelings about the wine, is accompanied by words that will direct the wine to that segment who want those wines and away from the rest of us. Words are inadequate. So are rating systems.

Putting the two together helps, but subjective is always going to be subjective. But just like democracy, it may not be perfect but its the best we have for now.

Part 1 of 3: Trying to convey the character of a wine
by Bob Henry
Posted on:1/23/2017 1:10:44 AM

Self-evidently this blog has balked at my too numerous quotes, so I will break them up into three discrete comments.

First one . . .

Excerpt from Jancis Robinson, M.W. Website

(circa 2002):

“How to Score Wine”

Link: http://www.jancisrobinson.com/articles/how-to-score-wine?layout=pdf

"So, I limp along with points and half-points out of 20, which means that the great majority of wines (though by no means all) are scored somewhere between 15 and 18.5, which admittedly gives me only eight possible scores for non-exceptional wines -- an improvement on the five star system but not much of one. (I try when tasting young wines to give a likely period when the wine will be drinking best, so I do cover the aspect of its potential for development.)"

[Robinson later re-invented her 20 point scale when she sampled mind-blowing older Yquems . . . taking her scale up to 26 (sic) points.]

Part 2 of 3: Trying to convey the character of a wine
by Bob Henry
Posted on:1/23/2017 1:12:30 AM

Part Two . . .

Excerpt from Wine Times (circa 1989) interview with Robert Parker:

". . . The newsletter was always meant to be a guide, one person's opinion. The scoring system was always meant to be an accessory to the written reviews, tasting notes.   That's why I use sentences and try and make it interesting.   Reading is a lost skill in America.   There's a certain segment of my readers who only look at numbers, but I think it is a much smaller segment than most wine writers would like to believe.   The tasting notes are one thing, but in order to communicate effectively and quickly where a wine placed vis-à-vis its peer group, a numerical scale was necessary.  If I didn't do that, it would have been a sort of cop-out."

Part 3 of 3: Trying to convey the character of a wine
by Bob Henry
Posted on:1/23/2017 1:14:08 AM

Part Three . . .

Excerpt from Slate

(posted June 15, 2007):

“Cherries, Berries, Asphalt, and Jam.

Why wine writers talk that way.”

Link: http://www.slate.com/articles/life/drink/2007/06/cherries_berries_asphalt_and_jam.html

By Mike Steinberger

"In his book The Taste of Wine, legendary French oenologist Emile Peynaud elegantly explained the conundrum. 'We tasters feel to some extent betrayed by language,' he wrote. 'It is impossible to describe a wine without simplifying and distorting its image.' This linguistic failure is surely one reason that numerical scores for wines have proven so popular; points are simplistic and distorting, too, but they at least give you something to hold onto -- more so than, say, 'spice box,' 'melted asphalt,' or 'liquefied minerals.'"

And one more quote . . .
by Bob Henry
Posted on:1/23/2017 1:19:07 AM

From The San Francisco Chronicle “Food & Wine” Section Letters-to-the-Editor

(June 22, 2007, Page Unknown):

“Keeping Score on Ratings”

Link: http://www.sfgate.com/wine/article/LETTERS-TO-WINE-Keeping-score-on-ratings-2572416.php

Editor -- Re [article titled] "Are Ratings Pointless?" (June 15, 2007):

Well, gee, all this fuss (yet again) about the 100-point scale. So it doesn't tell everything you need to know about a wine. Do four stars? Twenty points? No, you have to read the critic's words.

Does it reward subtle wines? That's up to the critic using the scale. Do subtle wines that lack drama ever get four-star ratings from the [San Francisco] Chronicle's team? No, they get two or 2½ stars, which pretty much corresponds to a 100-point score in the 80s.

The main reason I like to use the 100-point scale is that it lets me communicate more to my readers. They can tell that I liked a 90-point wine just a little better than an 89-point wine, but a 94-point wine a lot more than one rated 86. Doesn't that say more than giving both the 90-and 94-point wines three stars and both the 89- and 86-point wines 2½ stars?

(signed)

HARVEY STEIMAN

Editor at Large, Wine Spectator


You Broke It
by Charlie Olken
Posted on:1/23/2017 1:28:58 AM

Bob--

I am not sure what you are doing that clogs up the system, and I have no idea if or how this response to you will show up.

I am guessing that your are cutting and pasting, and our system seems to not like that.

If this were Twitter, you could be the Donald Trump of the CGCW blog. No disrespect meant, but we can't call this a Tweet Storm so perhaps it is just a blitz.

And now I have to cross my fingers that my words will not go off into the ether never to be seen again.

David Morrison's sequel blog
by Bob Henry
Posted on:2/9/2017 8:09:50 PM

David Morrison's follow-up piece on the Judgment of Paris:

"Why We No Longer Have The Data From The Judgment of Paris"

http://www.academicwino.com/2017/02/no-data-from-judgment-of-paris.html/?utm_source=feedburner&utm_medium=email&utm_campaign=Feed: TheAcademicWino (The Academic Wino)

Leave a comment below, but please limit your comments to 1,200 characters or less. We find it helpful to make a copy of our comments to be sure that they fit. In that way, you can edit them if they run long.

(Please note: your e-mail address will not be visible after posting)

Name
Email
Subject

 

Note: Refresh your browser to see your latest comments.

Having technical problems with the comment system? Click here.