Commentary: Two Types of Sensory Panels or Are There More?

Scriven's article is very timely in the sense that it addresses issues which we, as sensory professionals, take for granted when in fact they are not clear or have never been properly addressed. Her focus in terms of what is being measured, rather than who does the measurement, is particularly relevant. In Fig. 1 of Scriven's article the difference between trained and naïve assessors is shown in terms of accuracy. When there is a high level of accuracy, the panel size can be small (e.g., 8–15 assessors) and when there is a low level of accuracy, the panel size needs to be larger (e.g., 30–100 assessors). To further illustrate this point, a simulation can be performed: To simplify, suppose only two products, A and B, are to be scored on a given descriptor, e g., garlic, using a 0–100 sensory scale. The panel is formed with N assessors, out of which R assessors can discriminate correctly between the garlic levels of samples A and B. Suppose that the R assessors who discriminate correctly use the scale differently. For the A samples we assign random scores between 0 and 49 (low level of garlic flavor), and for the B samples we assign random scores between 51 and 100 (high level of garlic flavor). We suppose that the remaining (N–R) assessors cannot discriminate correctly between the garlic levels. We assign random scores between 0 and 100 for both the A and the B samples. Having assigned the scores we perform a pairwise t-test between A and B samples. For each combination of N and R we perform 500 simulations and calculate the average significance level (α) of the pairwise t-tests, choosing the value of R which gives the highest value of α < 5%. Results of percent discriminators necessary to obtain <5% significance level in the pairwise t-test for different numbers of total assessors are in Table 1. If we can assume that 16% of the population can perform just as well as a trained assessor for a given descriptor, e.g., garlic, and we randomly select 200 consumers, they will, as a whole and thanks to the 16% who discriminate correctly, perform just as well as a panel of 12 assessors with two-thirds of their members discriminating correctly. Thus, we can ask naïve consumers the same questions as a trained panel, the difference being in the number of respondents we need to obtain meaningful answers. Of course, for very analytical terms like “lipolysis” flavor in milk, only trained assessors can perform adequately. In Scriven's Table 2, it is indicated that for consumer panels, a location with some degree of being a realistic environment in which the product would be used is desirable. The most “realistic” situation for tasting beer could be in a bar, with friends, heartily discussing last night's football match. If the consumer drank two mugs of beer, and was later asked which he/she liked best, the first one or the second one, it is very likely that the answer would be: “They both tasted great!” But maybe once a month our beer consumer sits down to a beer at home and really tastes the product. This is when he/she decides that Brand X is good or poor and this will influence what brand he/she asks for at the bar. So perhaps an extremely realistic tasting situation is not always the most adequate. Scriven indicates that a sample size of between N = 50 and 100 is sufficient to find significant differences between products. We have an article in press (Hough et al. 2005) which presents a table for estimating the number of consumers necessary for liking tests for different values of Type I error, Type II error and size of difference. The standard deviations were obtained from consumer studies in five different countries. Considering the average standard error, an alpha value (Type I error) of 5%, a beta value (Type II error) of 10% and a difference between sample means of 10% of the sensory scale, gave an N value of 112 consumers, a value similar to Scriven's higher limit of N = 100. Looking at Scriven's Table 5 reminded me of Johnson's (1999) interesting comments on sample size. He stated that “unimportant differences that turn out significant are annoying, and important differences that fail statistical detection are truly depressing.”Table 2 (adapted from Johnson [1999]) shows interpretation of sample size as related to results of a statistical significance test. Quoting Johnson (1999): “The annoying unimportant differences that were significant indicate that too large a sample was obtained. Further, if an important difference was not significant, the investigator concludes that the sample was insufficient and calls for further research. This schizophrenic nature of the interpretation of significance greatly reduces its value.” The alternatives to significance testing proposed by Johnson (1999) were: estimates of effects and confidence intervals, decision theory and Bayesian approaches.