School of Mathematics and Statistics, University of Sheffield, UK.
Publication details: Submitted to The American Statistician, 2018.
Expert opinion and judgement enter into the practice of statistical inference and decision-making in numerous ways. Indeed, there is essentially no aspect of scientific investigation in which judgement is not required. Judgement is necessarily subjective, but should be made as carefully, as objectively, as scientifically as possible.
Elicitation of expert knowledge concerning an uncertain quantity expresses that knowledge in the form of a (subjective) probability distribution for the quantity. Such distributions play an important role in statistical inference (for example as prior distributions in a Bayesian analysis) and in evidence-based decision-making (for example as expressions of uncertainty regarding inputs to a decision model). This article sets out a number of practices through which elicitation can be made as rigorous and scientific as possible.
The principal focus is on the cognitive biases that experts are prone to when making probabilistic judgements, individually or in groups, and on how these can be addressed and minimised by a well-designed elicitation protocol. This is illustrated by the SHELF protocol.