The J. Miller Blind Tasting Theory


For my next blog post I am analyzing whether Jay Miller’s refusal to taste wines blind adversely affected his ability to objectively analyze wines tasted for WA reports. To do so I created (and am expanding) a spreadsheet of various wines he tasted during his stint with the Advocate from each region, such as Oregon, Washington, Argentina, Chile, Spain, and Australia. I am then comparing those to scores given to the same wines by WS critics when tasted blind. The goal is to see if he consistently rated wines higher than his WS counterparts, and whether his scores varied to a similar degree. My hypothesis is that his scores will vary less because he knows what wine he is drinking, and therefore goes into his tasting with a preconceived notion of quality and consistency.

So, I ask you, fair Berserkers:

What wines do you suggest I compare? So far I have tried to analyze wineries with varying styles, such as Brick House v. Domaine Serene, as well as wines in various price ranges, such as Numanthia Numanthia v. Numanthia Termes to see if his taste by producer affected his perception in quality.

What do you think I can do to make my analysis more informative, more accurate, etc. No promises that I’ll take your advice, but I am open to ideas to help control quality.

I don’t know. Comparing one reviewer to many may be stretching it. But in any case…


You do realize that few critics taste blind,yes? The Wine Spectator purports to. Tanzer does not IIRC nor do most of the WA critics. Any onsite tastings aren’t blind… you’re at the winery after all. So… how are you going to find a control? And how will you calibrate things to eliminate other factors that influence scores, most notably different preferences?

Great idea :slight_smile:

In terms of bottle selection, finding ones that are rated every (or almost every) vintage by both publications would be useful from a stats perspective. Also, your approach of comparing first vs second labels from a single winery is intriguing, as the “perceived quality” will definitely impact JM while WS will be agnostic to the hierarchy.

First ones that come to mind:
Quilceda Creek
Don Melchor
Viu Manent Viu 1

to whack the preference issue I am selecting massively different styles within the same region. To control for blind I selected only WS critics and blind test scores. WS largely discloses when wine is tasted non blind. Moreover, I used multiple critics at WS to get a larger sample of how tasting blind impacts standard deviations across the board.

hmm… intersting. Not sure how I feel about the preference approach. It seems to me that it will be impossible to separate out Jay’s preferences and their effect on his scores from the blind issue. The other problem you have is that you’re forced to assume that everyone means exactly the same thing by each point score from 85 to 100. I’m skeptical, esp since Jay’s no longer a critic (so I don’t see the point in assessing his performance) but… your time :slight_smile:

Using a standard deviation comparison as opposed to straight point comparison should provide a more accurate picture and prove/disprove the hypothesis.

Gotta do something between 9:00pm and 11:00pm (kids asleep before I’m asleep) while I’m drinking a mediocre 2000 Potel! Neener!

No worries. Every bottle is an education, and it’s not like I didn’t drink/enjoy it!

Trying to correlate one taster’s scores to another I believe throws too many variables into the mix to yield any useful data. If you had enough data, I believe you would prove that scores by themselves mean absolutely nothing, but that’s another conversation.

I’d say forget about looking at certain wines, and approach this by simply correlating score to price within certain categories, then see if non-blind tasting has a tighter correlation than blind. In this approach, price could be thought of as a type of non-blind score given by the market, and provides a kind of standard of measure by which scores for different wines could be compared to each other. The methodology becomes simpler and I think the results easier to understand (and explain to others).

I think it would be hard to have such analysis tell you anything.

Perhaps use only wines that you have 5 years of scores from Wine Advocate. Doesn’t have to be only Miller’s scores since not much WA tasting is blind. See how much variation there is. Do the same with wines that you have 5 years of scores from Wine Spectator.

If you discover more variation over time in WS scores, you could interpret that as evidence that non-blind tasting influences scoring. Or not.

You do know that last year he tasted a considerable amount of the Washington (and I think Oregon) wines blind and said so?

Blind at the winery, or plain blind? His article on 2009 Oregon suggests he tasted with winemakers and at wineries. Putting a brown bag on a Bergstrom and not knowing which vineyard it is from is hardly blind, and would hardly have any impact on whether he was affected by a bias or preconception. We do know that he wasn’t tasting Spanish wines blind, and that the VAST majority of his tastings were non-blind, and producer by producer.

Randy, that’s just about exactly what I am doing, but limiting the wines tasted to those tasted by Jay Miller. There is so much controversy around him right now that I am interested to see if much of it is based on hype, as opposed to results. So, I want to look specifically at him vs. his WS counterparts. While I understand that 1 critic v. multiple critics may create a lack of consistency, the goal is really to see if non-blind tasting precludes a wide variation in scores. The deviation from the median is what I want to know. That deviation is more important than the actual number-score to me.

Larry, your price analysis suggestion is really interesting. I might take a look at that and run a second comparison. I think that is a really good idea.

Mr. Kennedy, thanks for helping come up with the idea for the analysis, and I like your very-telling suggestion re the Viu Mananet!

Ah I see what you’re doing. Interesting approach.

I think all or most of us would suspect very strongly that non-blind tasting does promote less variation in scores, whether in our own cases or those of professional critics.

I’m not sure you’re going to have the right samples of data to prove that empirically with just WA and WS (and considering the 85 point floor that WA uses), but I’m always interested to see some number crunching.

Oops. I misread your hypothesis. If it is that JM consistently overrates wine (as compared to WS) for whatever reason, the answer is a resounding “yes”. Here’s a couple data points in my 5 minute review. I included everything I found where both a JM and WS review existed. In no case did JM review a wine and score it lower or equal to the WS review. This is precisely why I ignore JS reviews. Well, this score inflation and his craziness on drinking windows:). FWIW, I normally notice JM’s grade inflation on wines hovering around 90 points, where he scores it 90-92, others score 87-88. Guess which review your favorite online retailer uses?:). Which doesn’t bother me until the retailer turns a JM score into an RP score (e.g. “92 points from WA/Robert Parker”)


2006 Don Melcor 95 94
2007 Tarapaca Zavala 91 88
2007 El Nido 96 94
2007 Yering Pinot 94 89
2006 Long Shadows Pedestal 94 93
2006 Quilceda Cab 99 94
2007 Argyle Spirithouse 93 90

How did this become the J Miller theory when nearly every tasting note from every reviewer, professional and amateur, is from a non-blind tasting?

Agreed. I think the working hypothesis should be modified to “Jay Miller overscores wines”, dropping the “because he doesn’t blind taste” part of the sentence.

It’d be interesting to see if blind tasting was at the root of Laube’s frightening variability:). I suspect not.

As posted before, I don’t know of any influential critics who taste blind. I probably am one of the best at blind tasting but I agree that the blind tasting is not the answer. The above results seem to indicate that his ratings are pretty consistent with the WS’s ratings. BTW, I have heard that Jay Miller has a good palate and have no reason to doubt that.