The Critic vs. the Computer: A case of perceptual discrepancy

 

Did you know that I prefer organic wines to non-organic wines? I didn’t, either. But then I read this new paper from the American Association of Wine Economists, entitled “Does Organic Wine taste better? An Analysis of Experts’ Ratings,” and I found out that, yup, I do.

Well, kinda sorta. See, the paper’s authors decided to study “data from the three influential wine expert publications: Wine Advocate, Wine Enthusiast, and Wine Spectator,” and as it turned out, “During our period of study [74,148 wines produced in California between 1998 and 2009], the main tasters for California wines for Wine Advocate, Wine Enthusiast and Wine Spectator were Robert Parker, Steve Heimoff, and James Laube, respectively.”

The big P-H-L! They took our scores, crunched them in that esoteric way only economists can, and lo and behold, “Our results indicate that the adoption of wine eco-certification has a statistically significant and positive effect on wine ratings.”

How much? Not a lot: “Being eco-certified,” the authors found, “increases the score of the wine by 0.46 point on average.”

Well, one hardly knows where to begin. Right off the bat, I have a problem when the lesson that people will take away is that P-H-L (and by extension major critics) prefer organic wines to non-organic ones. Less than half a point difference? I suppose if they fed 74,148 scores into a computer and found a 0.46 point difference, then who am I to argue with HAL? But a 0.46 point difference doesn’t seem like very much to me. It’s not even round-uppable to the higher score (87.46 rounds down to 87).

But wait, there’s more. The following factors also had an impact on the scores of organically-certified wines, according to the paper:

• ” a 1% increase in the number of cases will decrease score by 0.003

point.”

• ” An increase in the number of years of certification experience by one [winery] decreases score by 0.09 point.”

Confused? I am. So the more cases wine the winery produces, the lower the score is; but the longer the winery has been certified organic, the lower the score also is!

How about the winemaker’s hair color? Did they include that?

The authors also counted the number of words in each review and found this: “Next, we examine the impact that eco-certification has on the number of words used in wine notes. As shown in regression (1) of Table 6, wine notes of eco-certified wines are not significantly longer than those of conventional wines. However, as shown in regressions (2) and (3), eco-certification increases the average number of positive words by 0.4 but has no statistically significant impact on the number of negative words.”

My interpretation of this is that it’s gibberish. The authors compiled a list of words [Table 7] but I don’t understand how they infer whether their use is positive or negative. Is “jammy” positive or negative? Do Parker, Laube and I even use it in the same way? How about “offbeat”? Is that good or bad? And “peat”: if I tasted that in an Islay Scotch it would be good, but in a Chardonnay?

The authors also state something that I don’t think is objectively true, or, even if it is, is irrelevant. “Second, as a related point, wine experts have a better knowledge about wine eco-certification and are able to differentiate between different types of eco-labels, namely organic wine and wine made with organically grown grapes, which represent different wine production processes with different impacts on quality.”

I’m not going to sit here and tell you I know the difference between different types of eco-labels. There are so damn many (different certifying agencies, “natural,” biodynamic, etc.), I get confused—and, while I’ll let Parker and Laube speak for themselves, I bet they get confused, too. Besides, if “All the publications claim blind review,” as the paper’s authors write, then we critics don’t even see the labels when we’re tasting and reviewing (much less would we have a tech sheet in front of us).

But finally, this statistic seems to be to be the last nail in the coffin of the study: “On average, 1.1% of the wines in the sample are eco-certified.” By my calculations, that’s a little over 800 wines—out of 74,148. I fail to see how you can extrapolate any useful information from such a small sample, compared to the huge number of wines in the study. Apples and oranges.

I’m no economist, it goes without saying. If I were, I guess I’d spend my days crunching numbers and coming up with interesting factoids. But I have to say, I don’t see the point of this particular study—not if it’s going to be used to make a claim that I don’t regard as true. For the record, let me say that I do not think organic wine is better. And you know what? I don’t care what the numbers say.