OK - ads don’t influence reviews - especially with blind tastings.
wine-econ.org" onclick="window.open(this.href);return false;
Who are [or were] the Wine Advocate “advertisers”?
Good question. A very intriguing differential for WA!
Ive never felt their ratings were biased. The type of wines their reviewers like is another topic altogether though.
Why is the Wine Economics study based on nine year old data points?
“Advertisers” always refers to WS advertisers, “WA advertisers” means WA scores for wines advertised in WS.
The original paper finds that wines advertisers score slightly lower in WS, on average, than wines from non-advertisers. However, it may be a result of the wines from advertisers being lower in quality. So, the author compared wines scored by both WS and WA, trying to use the WA scores as an indication of quality. It turns out the wines from producers who didn’t advertise in WS scored higher than wines from advertisers by a larger amount in WA than in WS. So, one might infer that WS has a bias for wines from their advertisers, partially compensating for their lower quality compared to wines from non-advertisers. Much statistical analysis follows.
I am reminded of the quote from Aristotle’s Ethics, “It is the mark of an educated mind to rest satisfied with the degree of precision which the nature of the subject admits and not to seek exactness where only an approximation is possible.”
-Al
Thanks for the explanation. Petrus, DRC and Screaming Eagle don’t take out a lot of full-page spreads, do they? Advertised wines tend by nature to be relatively large production products.
The number is perfectly skewed. Wines like Petrus, DRC and SE that don’t have to advertise help bring the weighted average up. Time tested wineries like say Calera, Montelena and Ridge get bitch slapped with ratings like 86 or 85 (which oddly come close to the average, as opposed to the 69s these wines used to get) and we are supposed think there is no bias? (I think we can all agree that the top cuvees for these wineries are at least excellent, therefore earning 90+ points)
I’m sorry, but if Diamond Creek or Montelena were going to produce and release that craptastic of a bottle of wine, all the critics would be saying so, not just one.
Ian
I disagree. Wine is all about personal likes and dislikes.
These reviewers are human. The Wine Spectator does the best job possible of removing the “human” element by taking the label and tasting blind. All you are left with is what is in the glass.
Clearly, certain styles and wines do not appeal to certain critics. But the same can be said of all reviewers, critics, retailers and consumers.
You just hope that each critic is doing the best that they can, and in an honest fashion.
The wines in question, from 9 years ago, were rated, on average, higher at the Wine Advocate than the Wine Spectator. The advertiser’s wines scored slightly less than non advertised wines, in this small sample set of 700+ wines.
Using the Wine Advocate as the control for this study is quite strange, especially amidst all of the allegations of impropriety recently.
Here are a few other comments.
The correlation coefficient between WS and WA scores was 0.43. It’s positive, which means the general tendency was for them to like the same wines. But it’s well less than 1, meaning there isn’t a strong correlation with the scores.
The main reason the WA scores were a little higher is that they rarely publish scores lower than 85 (as frequently noted). This skews the average for WA. It also complicates the statistical analysis.
The paper inferred that WS tended to score advertiser wines 1 point higher than expected from the statistical analysis. It also inferred that part of the reason is that WS was more likely to re-taste wines from advertisers if they scored low. But the number of wines that were re-tasted was fairly small, so this effect accounted for no more than 0.5 points. The paper could not explain the other 0.5 points, given the assumption that WS tastes blind. The final conclusion of the paper was that any bias for advertisers from WS had a fairly small effect on their scoring system.
I didn’t notice any error analysis. Even if you assume there is some fundamental characteristic of wines that can be reduced to a quality score, there is an uncertainty to the “measurement” represented by a critic’s score. It’s not meaningful to discuss a difference between average scores unless you can quantify the uncertainties in the measurements and the analysis (and the difference is larger than the uncertainty). If you and I measure the distance between Palo Alto and San Francisco with a ruler and I measure a distance that’s greater by centimeter, it doesn’t indicate that I must have a bias that San Francisco must be further away.
So, that was the reason I quoted Aristotle. You can calculate statistical quantities to any precision you like, but it doesn’t mean they are significant. That’s true even when you are trying to measure fundamental quantities, but I I think it’s even more true when you are analyzing perceptual differences.
-Al
Al
Presumably, they only used wines for ths tudy that were rated by BOTH WA and WS, so the fact that WA may not publish many scores under 85 points would be irrelevant to this study, correct?
I think the idea that blind tasting removes the “human element” is just another misplaced expectation of the tasting method. Tasting blind introduces many variables, including the context of the other wines in the lineup, but most importantly it actually emphasizes the taste preferences of the taster. By removing a lot of important information (such as how did this wine taste in the past), the method actually emphasizes the “human” element and the results are a quite skewed toward how a given person perceived the wine at a given moment of its evolution. The WS editors have easily characterized styles preferences (some of which are fairly extreme) and if anything tasting blind emphasizes these personal preferences.
Presumably, they only used wines for ths tudy that were rated by BOTH WA and WS, so the fact that WA may not publish many scores under 85 points would be irrelevant to this study, correct?
Actually, no. The WS and WA scores aren’t very strongly correlated so there tends to be a spread in their scores for the same wines. There will be wines that WA scored 87 and WS scored 83 (for example), but there are very few wines where the scores are reversed. Putting it another way, there will be plenty of wines that WS scored below 85 even among the group of wines that WA also tasted. But there will be very few wines that WA scored below 85. So, the WA average will be higher than it would be if they published the scores below 85.
-Al
Kevin, I’d agree that tasting blind isn’t necessarily better than tasting non-blind. A lot depends on the taster and the circumstances. I know that when making a wine it can be important to have experience with the vineyard and with similar vintages. So, I’ve never much cared whether critics were tasting blind except in those instances where I thought they had an agenda.
-Al
I’d like to reemphasize Dan’s point about the human aspect of having the same reviewing, with the same preferences, the same palate, doing the same region year in and year out. I agree with Kevin that tasting blind can’t remove that human element. As a result, it’s always good to note reviews are done most frequently by individuals who have a taste as singular as each of our own. To me, that makes wine reviews not objective benchmarks of quality or greatness but merely a useful data point from which to calibrate your palate to that of another.
Tom and Kevin make good points.
The bottom line is you ought to calibrate with a reviewer, assuming them to be credible.
If you know that Laube hates Montelena and you love Montelena, then just skip over those reviews.
That being said, it really is what is in the glass, without the romance. As I have said before, if a wine critic drives up to Harlan, or Ch Margaux to taste and review wine, then they already have a preconceived notion as to how many points are in the glass. Bias has set in.
Al,
I think I see what you are saying.
The “control” was really the WS wines that contained a WA review.
Since WA rarely publishes scores below 85 points, if WS had scored something 96 and WA scored it “unpublished,” then it never made it into the study.
Is that correct?
Absolutely agree on this point, Kevin. Preaching to the choir (of one) here IMHO.
There seems to be this belief that tasting blind guarantees “objectivity” and makes for a more valid TN. In my case, I find my
TN’s from blind tastings mch less accurate than non-blind.
Tom
Tom
What does “less accurate” mean?
Daniel, that’s right. Some of the wines that WA scored lower than WS didn’t make it into the study because WA didn’t publish the score.
-Al
Daniel,
Oftentimes, people will bring a mystery wine to my wine tastings. When I’m tasting the wine blind,
I’m usually struggling to identify the vintage/variety/origin, I find my descriptors are not very precise in the TN.
I will oftentimes latch onto one facet of the wine and be totally led astray. Blackberries?? Gotta be a Syrah w/ maybe
10 yrs of age on it. Nope…a 3 yr old RRV Zin. If it’s something like a Refosco in a Syrah tasting, when I go back and
taste it, knowing it’s a Refosco, I find my descriptors are just more accurate in describing the wine.
In a perfect world…or if I were a professional critic…my TN would, in theory, be the same. Alas…I’m just
a rank amateur when it comes to tasting wine. That’s why I’m not a big believer in blind tastings. Just knowng the
varity allows me to taste w/ much more precision, I feel.
Tom