Quantitative v/s Qualitative Evaluations: Impact Factors and Wine Experts

Personally, I react to the importance attached to impact factors of all varieties, h-indices and their cousins, and metrics such as the pure number of research citations or papers published, with an emotion ranging between dismay and disgust.

I think they are a lazy substitute for actually reading a person's research and evaluating its worth individually. While it is fashionable, and getting increasingly so, I've never really been a big fan of using purely quantitative factors to measure the worth of an individual, university, or country.

You wouldn't necessarily think that the musician who sells the most records, or has the most covers made is necessarily the best (that would rate the likes of Back Street Boys over bands like Dream Theater).

Not that I don't understand the perils of subjectivity.

I recently came across this Wall Street Journal article on subjective assessment of wine experts through this blog. It highlights the problems of purely subjective evaluations. From the article:

In France, a decade ago a wine researcher named Fréderic Brochet served 57 French wine experts two identical midrange Bordeaux wines, one in an expensive Grand Cru bottle, the other accommodated in the bottle of a cheap table wine. The gurus showed a significant preference for the Grand Cru bottle, employing adjectives like "excellent" more often for the Grand Cru, and "unbalanced," and "flat" more often for the table wine.

Another similar story:

Francesco Grande, a vintner whose family started making wine in 1827 Italy, told me of a friend at a well-known Paso Robles winery who had conducted his own test, sending the same wine to a wine competition under three different labels. Two of the identical samples were rejected, he said, "one with the comment 'undrinkable.' " The third bottle was awarded a double gold medal.

I've never really had a very discriminating sense of taste (which by the way, I always viewed as an asset, since cheap works just as well as expensive). In the world of "wines" there are so many rules. The wine-nazis decide which vintage of which wine goes best with what kind of food. They can apparently discriminate all the nuances of a sophisticated wine. Interestingly from the article:

For example, a 1996 study in the Journal of Experimental Psychology showed that even flavor-trained professionals cannot reliably identify more than three or four components in a mixture, although wine critics regularly report tasting six or more. There are eight in this description, from The Wine News, as quoted on wine.com, of a Silverado Limited Reserve Cabernet Sauvignon 2005 that sells for more than $100 a bottle: "Dusty, chalky scents followed by mint, plum, tobacco and leather. Tasty cherry with smoky oak accents…" Another publication, The Wine Advocate, describes a wine as having "promising aromas of lavender, roasted herbs, blueberries, and black currants." What is striking about this pair of descriptions is that, although they are very different, they are descriptions of the same Cabernet. One taster lists eight flavors and scents, the other four, and not one of them coincide.