CT vs the Wine Critics (or: oh look, we’re not idiots)

Fun article deploying some basic analysis about score correlation between CT and major wine critics.

Some of the Vox conclusions:

“Ultimately, we think our analysis is very supportive of community wine reviews. If non-professional wine enthusiasts truly were lacking in knowledge or expertise, we would expect to see little or no correlation with professionals. We saw just the opposite. [snip]

In the end, however, CellarTracker offers two compelling advantages. The first is price. … The second is breadth. For California alone, CellarTracker covers more wines than all the critics we examined combined.”

LOL! This is some laughable “math.”

Interesting article. I bet you could get the correlation even higher with some minor tweaks:

1. Drop all CT scores with only one reviewer.

2. Drop all scores below 85. It’s not really a 100 point scale.

Question is how much bias there is where a CT user could be inadvertently imtating a critic’s score.

Bingo. I think it’s silly to assume a priori that just because CT aligns with the critics that CT is necessarily right. I agree that it’s more likely that CT contributors are being anchored by scores that they’ve seen from the critics

I have a simple hypothesis which might explain why the CT scores correlate so well with the critics’: a lot of CTers are aware of the critics’ scores and notes before they enter their own in CT. I have no way of testing this hypothesis.

3. Remove all tasting notes that are more than X years from the wine’s release to better coincide with when the “Pro” did their tasting & scoring.

The outlier mentioned in the article is 1999 Testarossa Chardonnay Sleepy Hollow, and Keith Levenberg tasted it in 2009- 10 years after the vintage.

This is the obvious issue for me. Look at this 97 point cab I bought. I’m opening this 97 point cab. Oh, it’s so good. This is what a 97 pointer tastes like. 98 points for me because it’s evolved during the 8 months post-release.

Eric LeVine is establishing wine world domination. Fantastic!

I love what Eric has done with CT, but the article seems to say “No wonder the critics liked it, look what the community said on CT.” Previous comments about comfirmation bias are likely to happen in many cases. I have said it here before but unless someone is very close to the ground with winemakers to get access to a small circle of new emerging wines by word-of-mouth, the vast majority of the other 1% of wine buyers learns about something new from a sommelier, retailer or writer/critic - people who spend a career bringing new, exciting releases to the attention of their customers. Fundamentally, there is always someone else who has tasted a wine before the public gets access (apart from the die-hard fanatics). I look at CT to see how an older wine is tasting, not to find new producers to go review.

This is obviously the answer.

I have read this article before. I believe what they are trying to say is:

\

1. CT community scores align more with the critics scores than one would assume. Remember also that this article was written before Vivino, Delictable et al had a large enough data base to make it worth using them also. It would be interesting to see how they now align with the critics and each other.

2. They state that the one low score only had one review for that wine.

3. Notice the difference between the critics themselves, that tends to be higher than the difference with CT.

My take on the article was that the community reviews are just as reliable a indicator of a wines quality as the critics and you don’t have to pay the community to access the reviews unlike the critics.

Just my 2 cents worth.

I’ve said for years that many user scores tend to drift towards the critics scores on CT. Folks don’t tend to like that and want to believe in the wonders of crowdsourcing. They tend to forget that crowds most often will follow things like the loudest person and big, bright shiny things such as big, popular critics scores.

It’s quite funny that they believe it so strongly that they can say that the crowdsourced scores clump around critics scores as if those critics are some objective measure.

My take on the article was that the community reviews are just as reliable a indicator of a wines quality as the critics and you don’t have to pay the community to access the reviews unlike the critics.

Um, isn’t that exactly what the article was saying?

And the posters noted a few reasons as to why that may be. So you can get the critics’ scores by reading the crowdsourced reviews, or you can get them unfiltered, but you’re kind of getting the same.

Except that are either really talking about a wine’s “quality”? More like simply crowd preferences.

And of course, the article misses the key value of CT. It’s not the scores! It’s all the review content and useful information about state of tannins, acid, balance, etc. I love CT reviews, unbelievably helpful for thinking about whether a wine is ‘ready’ for my particular taste.

Reviewers who are often tasting wines on or before release are remarkably unhelpful on this question.

Ground Hog Day

And if this is true, then to check the average of critics scores, just go to CT. Lots cheaper than subscribing to multiple rating services. The only reason to subscribe IMHO is to get written background on producers and details of estates. That is missing from CT and is probably THE major factor in my wine buying these days. I want to know the style, the philosophy, the flavor profile, etc. At this point in time just chasing scores seems pointless. Just means a high score enables producers jack up the price. Can’t blame them, but a 100 pointer gets immediately scratched off my buying list these days.

This. Well stated.

Also, as someone who reads and interprets statistical literature on a daily basis to guide patient care, this article is so ignorant of of how correlation works and stupid in interpretation as to make me wonder how it was ever published. Certainly not written by someone who understands either wine or statistics.

There is zero mention of how reviews are not a continuous variable (though an average of those reviews is). Jancis scores, when converted to 100-point scale, go in increments of 2.5, which, when compared to a continuous average, is likely the main reason for the discrepancy in Pearson coefficient.

And California high end wines, which are the ones that have both critic scores and CT scores, have their sales driven strongly by critic reviews, whether they’re 92-point shelf talkers for $30 wines or 98-point WA reviews for $300 wines. Most CT scores for these wines basically parrot the WA reviews, are written by folks who are justifying their own purchases and/or already highly subscriptive to groupthink, as others have noted. And California was the worst possible region to select for this study. They could easily have done a within-group analysis (inter-observer reliability) to see how much agreement there is between CT raters for the same wine, which is interesting.

This is a great point. It’s really the last 15 points that seem to matter. This begs the question if we really should adopt a scale from Spinal Tap. What would be more enticing, a 100 point wine or one that went to 11?

Quite!

7oR79ja1u-o

I’m 93 points on the article.