For my next blog post I am analyzing whether Jay Miller’s refusal to taste wines blind adversely affected his ability to objectively analyze wines tasted for WA reports. To do so I created (and am expanding) a spreadsheet of various wines he tasted during his stint with the Advocate from each region, such as Oregon, Washington, Argentina, Chile, Spain, and Australia. I am then comparing those to scores given to the same wines by WS critics when tasted blind. The goal is to see if he consistently rated wines higher than his WS counterparts, and whether his scores varied to a similar degree. My hypothesis is that his scores will vary less because he knows what wine he is drinking, and therefore goes into his tasting with a preconceived notion of quality and consistency.
So, I ask you, fair Berserkers:
What wines do you suggest I compare? So far I have tried to analyze wineries with varying styles, such as Brick House v. Domaine Serene, as well as wines in various price ranges, such as Numanthia Numanthia v. Numanthia Termes to see if his taste by producer affected his perception in quality.
What do you think I can do to make my analysis more informative, more accurate, etc. No promises that I’ll take your advice, but I am open to ideas to help control quality.
You do realize that few critics taste blind,yes? The Wine Spectator purports to. Tanzer does not IIRC nor do most of the WA critics. Any onsite tastings aren’t blind… you’re at the winery after all. So… how are you going to find a control? And how will you calibrate things to eliminate other factors that influence scores, most notably different preferences?
In terms of bottle selection, finding ones that are rated every (or almost every) vintage by both publications would be useful from a stats perspective. Also, your approach of comparing first vs second labels from a single winery is intriguing, as the “perceived quality” will definitely impact JM while WS will be agnostic to the hierarchy.
First ones that come to mind:
Quilceda Creek
Don Melchor
Viu Manent Viu 1
to whack the preference issue I am selecting massively different styles within the same region. To control for blind I selected only WS critics and blind test scores. WS largely discloses when wine is tasted non blind. Moreover, I used multiple critics at WS to get a larger sample of how tasting blind impacts standard deviations across the board.
hmm… intersting. Not sure how I feel about the preference approach. It seems to me that it will be impossible to separate out Jay’s preferences and their effect on his scores from the blind issue. The other problem you have is that you’re forced to assume that everyone means exactly the same thing by each point score from 85 to 100. I’m skeptical, esp since Jay’s no longer a critic (so I don’t see the point in assessing his performance) but… your time
Using a standard deviation comparison as opposed to straight point comparison should provide a more accurate picture and prove/disprove the hypothesis.
Trying to correlate one taster’s scores to another I believe throws too many variables into the mix to yield any useful data. If you had enough data, I believe you would prove that scores by themselves mean absolutely nothing, but that’s another conversation.
I’d say forget about looking at certain wines, and approach this by simply correlating score to price within certain categories, then see if non-blind tasting has a tighter correlation than blind. In this approach, price could be thought of as a type of non-blind score given by the market, and provides a kind of standard of measure by which scores for different wines could be compared to each other. The methodology becomes simpler and I think the results easier to understand (and explain to others).
I think it would be hard to have such analysis tell you anything.
Perhaps use only wines that you have 5 years of scores from Wine Advocate. Doesn’t have to be only Miller’s scores since not much WA tasting is blind. See how much variation there is. Do the same with wines that you have 5 years of scores from Wine Spectator.
If you discover more variation over time in WS scores, you could interpret that as evidence that non-blind tasting influences scoring. Or not.
Blind at the winery, or plain blind? His article on 2009 Oregon suggests he tasted with winemakers and at wineries. Putting a brown bag on a Bergstrom and not knowing which vineyard it is from is hardly blind, and would hardly have any impact on whether he was affected by a bias or preconception. We do know that he wasn’t tasting Spanish wines blind, and that the VAST majority of his tastings were non-blind, and producer by producer.
Randy, that’s just about exactly what I am doing, but limiting the wines tasted to those tasted by Jay Miller. There is so much controversy around him right now that I am interested to see if much of it is based on hype, as opposed to results. So, I want to look specifically at him vs. his WS counterparts. While I understand that 1 critic v. multiple critics may create a lack of consistency, the goal is really to see if non-blind tasting precludes a wide variation in scores. The deviation from the median is what I want to know. That deviation is more important than the actual number-score to me.
Larry, your price analysis suggestion is really interesting. I might take a look at that and run a second comparison. I think that is a really good idea.
Mr. Kennedy, thanks for helping come up with the idea for the analysis, and I like your very-telling suggestion re the Viu Mananet!
I think all or most of us would suspect very strongly that non-blind tasting does promote less variation in scores, whether in our own cases or those of professional critics.
I’m not sure you’re going to have the right samples of data to prove that empirically with just WA and WS (and considering the 85 point floor that WA uses), but I’m always interested to see some number crunching.
Oops. I misread your hypothesis. If it is that JM consistently overrates wine (as compared to WS) for whatever reason, the answer is a resounding “yes”. Here’s a couple data points in my 5 minute review. I included everything I found where both a JM and WS review existed. In no case did JM review a wine and score it lower or equal to the WS review. This is precisely why I ignore JS reviews. Well, this score inflation and his craziness on drinking windows:). FWIW, I normally notice JM’s grade inflation on wines hovering around 90 points, where he scores it 90-92, others score 87-88. Guess which review your favorite online retailer uses?:). Which doesn’t bother me until the retailer turns a JM score into an RP score (e.g. “92 points from WA/Robert Parker”)
Agreed. I think the working hypothesis should be modified to “Jay Miller overscores wines”, dropping the “because he doesn’t blind taste” part of the sentence.
As posted before, I don’t know of any influential critics who taste blind. I probably am one of the best at blind tasting but I agree that the blind tasting is not the answer. The above results seem to indicate that his ratings are pretty consistent with the WS’s ratings. BTW, I have heard that Jay Miller has a good palate and have no reason to doubt that.
![[popcorn.gif]](/uploads/db3686/original/2X/1/1ca9f29573458cca2424a66a50d6c1261081dd3e.gif "[popcorn.gif]"){kind=small}
